Linus provided a implementation for this for x2apic_cluster and mumbled at http://marc.info/?l=linux-kernel&m=144660325518209 : > Side note, I have an *old* patch that I think simplifies the > (reasonably common) case of sending IPI's to individual CPU's. That's > done by the "reschedule ipi" in particular. > > That's something that is trivially done on the x2apic, but that the > mask-based interfaces makes insanely complicated. > > I have *not* rebased this on top of modern kernels so it may not > actually work as a patch any more, because I'm just sending this out > as a "Hmm, what do you guys think" rather than a real submission. I thought it's worthwhile so I had a look at the other apic implementations and all but apic_default and apic_flat can be converted halfways simple to provide a single target callback. The following series does that and gets rid of the wrapper in the call site at the end. Compiles and didn't explode when booting it, so it must be perfect :) Thanks, tglx --- arch/x86/include/asm/apic.h | 1 + arch/x86/include/asm/ipi.h | 2 ++ arch/x86/kernel/apic/apic_flat_64.c | 19 +++++-------------- arch/x86/kernel/apic/apic_noop.c | 2 ++ arch/x86/kernel/apic/apic_numachip.c | 2 ++ arch/x86/kernel/apic/bigsmp_32.c | 10 +++------- arch/x86/kernel/apic/ipi.c | 18 ++++++++++++++++++ arch/x86/kernel/apic/probe_32.c | 1 + arch/x86/kernel/apic/x2apic_cluster.c | 12 ++++++++++++ arch/x86/kernel/apic/x2apic_phys.c | 12 ++++++++++++ arch/x86/kernel/apic/x2apic_uv_x.c | 1 + arch/x86/kernel/smp.c | 4 ++-- 12 files changed, 61 insertions(+), 23 deletions(-)
[-- Attachment #1: 0001-x86-add-a-single-target-IPI-function-to-the-apic.patch --] [-- Type: text/plain, Size: 2150 bytes --] From: Linus Torvalds <torvalds@linux-foundation.org> We still fall back on the "send mask" versions if an apic definition doesn't have the single-target version, but at least this allows the (trivial) case for the common clustered x2apic case. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/include/asm/apic.h | 1 + arch/x86/kernel/smp.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) Index: linux/arch/x86/include/asm/apic.h =================================================================== --- linux.orig/arch/x86/include/asm/apic.h +++ linux/arch/x86/include/asm/apic.h @@ -303,6 +303,7 @@ struct apic { unsigned int *apicid); /* ipi */ + void (*send_IPI)(int cpu, int vector); void (*send_IPI_mask)(const struct cpumask *mask, int vector); void (*send_IPI_mask_allbutself)(const struct cpumask *mask, int vector); Index: linux/arch/x86/kernel/smp.c =================================================================== --- linux.orig/arch/x86/kernel/smp.c +++ linux/arch/x86/kernel/smp.c @@ -115,6 +115,18 @@ static atomic_t stopping_cpu = ATOMIC_IN static bool smp_no_nmi_ipi = false; /* + * Helper wrapper: not all apic definitions support sending to + * a single CPU, so we fall back to sending to a mask. + */ +static void send_IPI_cpu(int cpu, int vector) +{ + if (apic->send_IPI) + apic->send_IPI(cpu, vector); + else + apic->send_IPI_mask(cpumask_of(cpu), vector); +} + +/* * this function sends a 'reschedule' IPI to another CPU. * it goes straight through and wastes no time serializing * anything. Worst case is that we lose a reschedule ... @@ -125,12 +137,12 @@ static void native_smp_send_reschedule(i WARN_ON(1); return; } - apic->send_IPI_mask(cpumask_of(cpu), RESCHEDULE_VECTOR); + send_IPI_cpu(cpu, RESCHEDULE_VECTOR); } void native_send_call_func_single_ipi(int cpu) { - apic->send_IPI_mask(cpumask_of(cpu), CALL_FUNCTION_SINGLE_VECTOR); + send_IPI_cpu(cpu, CALL_FUNCTION_SINGLE_VECTOR); } void native_send_call_func_ipi(const struct cpumask *mask)
[-- Attachment #1: x86-apic--Implement-single-target-IPI-function-for-clustered-apic --] [-- Type: text/plain, Size: 1412 bytes --] From: Linus Torvalds <torvalds@linux-foundation.org> [ tglx: Split it out from the patch which provides the new callback and wrapped it into local_irq_save/restore ] Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/x2apic_cluster.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) Index: linux/arch/x86/kernel/apic/x2apic_cluster.c =================================================================== --- linux.orig/arch/x86/kernel/apic/x2apic_cluster.c +++ linux/arch/x86/kernel/apic/x2apic_cluster.c @@ -23,6 +23,17 @@ static inline u32 x2apic_cluster(int cpu return per_cpu(x86_cpu_to_logical_apicid, cpu) >> 16; } +static void x2apic_send_IPI(int cpu, int vector) +{ + u32 dest = per_cpu(x86_cpu_to_logical_apicid, cpu); + unsigned long flags; + + x2apic_wrmsr_fence(); + local_irq_save(flags); + __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); + local_irq_restore(flags); +} + static void __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) { @@ -266,6 +277,7 @@ static struct apic apic_x2apic_cluster = .cpu_mask_to_apicid_and = x2apic_cpu_mask_to_apicid_and, + .send_IPI = x2apic_send_IPI, .send_IPI_mask = x2apic_send_IPI_mask, .send_IPI_mask_allbutself = x2apic_send_IPI_mask_allbutself, .send_IPI_allbutself = x2apic_send_IPI_allbutself,
[-- Attachment #1: x86-apic--Implement-default-single-target-IPI-function --] [-- Type: text/plain, Size: 1364 bytes --] apic_physflat and bigsmp_apic can share that implementation. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/include/asm/ipi.h | 1 + arch/x86/kernel/apic/ipi.c | 10 ++++++++++ 2 files changed, 11 insertions(+) Index: linux/arch/x86/include/asm/ipi.h =================================================================== --- linux.orig/arch/x86/include/asm/ipi.h +++ linux/arch/x86/include/asm/ipi.h @@ -119,6 +119,7 @@ static inline void native_apic_mem_write(APIC_ICR, cfg); } +extern void default_send_IPI_single_phys(int cpu, int vector); extern void default_send_IPI_mask_sequence_phys(const struct cpumask *mask, int vector); extern void default_send_IPI_mask_allbutself_phys(const struct cpumask *mask, Index: linux/arch/x86/kernel/apic/ipi.c =================================================================== --- linux.orig/arch/x86/kernel/apic/ipi.c +++ linux/arch/x86/kernel/apic/ipi.c @@ -18,6 +18,16 @@ #include <asm/proto.h> #include <asm/ipi.h> +void default_send_IPI_single_phys(int cpu, int vector) +{ + unsigned long flags; + + local_irq_save(flags); + __default_send_IPI_dest_field(per_cpu(x86_cpu_to_apicid, cpu), + vector, APIC_DEST_PHYSICAL); + local_irq_restore(flags); +} + void default_send_IPI_mask_sequence_phys(const struct cpumask *mask, int vector) { unsigned long query_cpu;
[-- Attachment #1: x86-apic--Remove-pointless-indirections-from-apic_physflat --] [-- Type: text/plain, Size: 1704 bytes --] No value in having 32 byte extra text. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/apic_flat_64.c | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) Index: linux/arch/x86/kernel/apic/apic_flat_64.c =================================================================== --- linux.orig/arch/x86/kernel/apic/apic_flat_64.c +++ linux/arch/x86/kernel/apic/apic_flat_64.c @@ -230,17 +230,6 @@ static int physflat_acpi_madt_oem_check( return 0; } -static void physflat_send_IPI_mask(const struct cpumask *cpumask, int vector) -{ - default_send_IPI_mask_sequence_phys(cpumask, vector); -} - -static void physflat_send_IPI_mask_allbutself(const struct cpumask *cpumask, - int vector) -{ - default_send_IPI_mask_allbutself_phys(cpumask, vector); -} - static void physflat_send_IPI_allbutself(int vector) { default_send_IPI_mask_allbutself_phys(cpu_online_mask, vector); @@ -248,7 +237,7 @@ static void physflat_send_IPI_allbutself static void physflat_send_IPI_all(int vector) { - physflat_send_IPI_mask(cpu_online_mask, vector); + default_send_IPI_mask_sequence_phys(cpu_online_mask, vector); } static int physflat_probe(void) @@ -292,8 +281,8 @@ static struct apic apic_physflat = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, - .send_IPI_mask = physflat_send_IPI_mask, - .send_IPI_mask_allbutself = physflat_send_IPI_mask_allbutself, + .send_IPI_mask = default_send_IPI_mask_sequence_phys, + .send_IPI_mask_allbutself = default_send_IPI_mask_allbutself_phys, .send_IPI_allbutself = physflat_send_IPI_allbutself, .send_IPI_all = physflat_send_IPI_all, .send_IPI_self = apic_send_IPI_self,
[-- Attachment #1: x86-apic--Implement-single-IPI-for-apic_physflat --] [-- Type: text/plain, Size: 736 bytes --] Use the default implementation. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/apic_flat_64.c | 1 + 1 file changed, 1 insertion(+) Index: linux/arch/x86/kernel/apic/apic_flat_64.c =================================================================== --- linux.orig/arch/x86/kernel/apic/apic_flat_64.c +++ linux/arch/x86/kernel/apic/apic_flat_64.c @@ -281,6 +281,7 @@ static struct apic apic_physflat = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = default_send_IPI_single_phys, .send_IPI_mask = default_send_IPI_mask_sequence_phys, .send_IPI_mask_allbutself = default_send_IPI_mask_allbutself_phys, .send_IPI_allbutself = physflat_send_IPI_allbutself,
[-- Attachment #1: x86-apic--Remove-pointless-indirections-from-bigsmp_apic --] [-- Type: text/plain, Size: 1364 bytes --] Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/bigsmp_32.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) Index: linux/arch/x86/kernel/apic/bigsmp_32.c =================================================================== --- linux.orig/arch/x86/kernel/apic/bigsmp_32.c +++ linux/arch/x86/kernel/apic/bigsmp_32.c @@ -96,11 +96,6 @@ static int bigsmp_phys_pkg_id(int cpuid_ return cpuid_apic >> index_msb; } -static inline void bigsmp_send_IPI_mask(const struct cpumask *mask, int vector) -{ - default_send_IPI_mask_sequence_phys(mask, vector); -} - static void bigsmp_send_IPI_allbutself(int vector) { default_send_IPI_mask_allbutself_phys(cpu_online_mask, vector); @@ -108,7 +103,7 @@ static void bigsmp_send_IPI_allbutself(i static void bigsmp_send_IPI_all(int vector) { - bigsmp_send_IPI_mask(cpu_online_mask, vector); + default_send_IPI_mask_sequence_phys(cpu_online_mask, vector); } static int dmi_bigsmp; /* can be set by dmi scanners */ @@ -180,7 +175,7 @@ static struct apic apic_bigsmp = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, - .send_IPI_mask = bigsmp_send_IPI_mask, + .send_IPI_mask = default_send_IPI_mask_sequence_phys, .send_IPI_mask_allbutself = NULL, .send_IPI_allbutself = bigsmp_send_IPI_allbutself, .send_IPI_all = bigsmp_send_IPI_all,
[-- Attachment #1: x86-apic--Implement-single-IPI-for-bigsmp_apic --] [-- Type: text/plain, Size: 686 bytes --] Use the default implementation. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/bigsmp_32.c | 1 + 1 file changed, 1 insertion(+) Index: linux/arch/x86/kernel/apic/bigsmp_32.c =================================================================== --- linux.orig/arch/x86/kernel/apic/bigsmp_32.c +++ linux/arch/x86/kernel/apic/bigsmp_32.c @@ -175,6 +175,7 @@ static struct apic apic_bigsmp = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = default_send_IPI_single_phys, .send_IPI_mask = default_send_IPI_mask_sequence_phys, .send_IPI_mask_allbutself = NULL, .send_IPI_allbutself = bigsmp_send_IPI_allbutself,
[-- Attachment #1: x86-apic--Implement-single-IPI-for-x2apic_phys --] [-- Type: text/plain, Size: 1170 bytes --] Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/x2apic_phys.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) Index: linux/arch/x86/kernel/apic/x2apic_phys.c =================================================================== --- linux.orig/arch/x86/kernel/apic/x2apic_phys.c +++ linux/arch/x86/kernel/apic/x2apic_phys.c @@ -36,6 +36,17 @@ static int x2apic_acpi_madt_oem_check(ch return x2apic_enabled() && (x2apic_phys || x2apic_fadt_phys()); } +static void x2apic_send_IPI(int cpu, int vector) +{ + u32 dest = per_cpu(x86_cpu_to_apicid, cpu); + unsigned long flags; + + x2apic_wrmsr_fence(); + local_irq_save(flags); + __x2apic_send_IPI_dest(dest, vector, APIC_DEST_PHYSICAL); + local_irq_restore(flags); +} + static void __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) { @@ -122,6 +133,7 @@ static struct apic apic_x2apic_phys = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = x2apic_send_IPI, .send_IPI_mask = x2apic_send_IPI_mask, .send_IPI_mask_allbutself = x2apic_send_IPI_mask_allbutself, .send_IPI_allbutself = x2apic_send_IPI_allbutself,
[-- Attachment #1: x86-apic--Implement-single-IPI-for-x2apic_uv --] [-- Type: text/plain, Size: 679 bytes --] The function already exists. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/x2apic_uv_x.c | 1 + 1 file changed, 1 insertion(+) Index: linux/arch/x86/kernel/apic/x2apic_uv_x.c =================================================================== --- linux.orig/arch/x86/kernel/apic/x2apic_uv_x.c +++ linux/arch/x86/kernel/apic/x2apic_uv_x.c @@ -406,6 +406,7 @@ static struct apic __refdata apic_x2apic .cpu_mask_to_apicid_and = uv_cpu_mask_to_apicid_and, + .send_IPI = uv_send_IPI_one, .send_IPI_mask = uv_send_IPI_mask, .send_IPI_mask_allbutself = uv_send_IPI_mask_allbutself, .send_IPI_allbutself = uv_send_IPI_allbutself,
[-- Attachment #1: x86-apic--Implement-single-IPI-for-apic_numachip --] [-- Type: text/plain, Size: 1047 bytes --] The function already exists. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/apic_numachip.c | 2 ++ 1 file changed, 2 insertions(+) Index: linux/arch/x86/kernel/apic/apic_numachip.c =================================================================== --- linux.orig/arch/x86/kernel/apic/apic_numachip.c +++ linux/arch/x86/kernel/apic/apic_numachip.c @@ -276,6 +276,7 @@ static const struct apic apic_numachip1 .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = numachip_send_IPI_one, .send_IPI_mask = numachip_send_IPI_mask, .send_IPI_mask_allbutself = numachip_send_IPI_mask_allbutself, .send_IPI_allbutself = numachip_send_IPI_allbutself, @@ -327,6 +328,7 @@ static const struct apic apic_numachip2 .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = numachip_send_IPI_one, .send_IPI_mask = numachip_send_IPI_mask, .send_IPI_mask_allbutself = numachip_send_IPI_mask_allbutself, .send_IPI_allbutself = numachip_send_IPI_allbutself,
[-- Attachment #1: x86-apic--Implement-single-IPI-for-apic_noop --] [-- Type: text/plain, Size: 994 bytes --] Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/apic_noop.c | 2 ++ 1 file changed, 2 insertions(+) Index: linux/arch/x86/kernel/apic/apic_noop.c =================================================================== --- linux.orig/arch/x86/kernel/apic/apic_noop.c +++ linux/arch/x86/kernel/apic/apic_noop.c @@ -30,6 +30,7 @@ #include <asm/e820.h> static void noop_init_apic_ldr(void) { } +static void noop_send_IPI(int cpu, int vector) { } static void noop_send_IPI_mask(const struct cpumask *cpumask, int vector) { } static void noop_send_IPI_mask_allbutself(const struct cpumask *cpumask, int vector) { } static void noop_send_IPI_allbutself(int vector) { } @@ -144,6 +145,7 @@ struct apic apic_noop = { .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and, + .send_IPI = noop_send_IPI, .send_IPI_mask = noop_send_IPI_mask, .send_IPI_mask_allbutself = noop_send_IPI_mask_allbutself, .send_IPI_allbutself = noop_send_IPI_allbutself,
[-- Attachment #1: x86-apic--Provide-default-send-single-IPI-wrapper --] [-- Type: text/plain, Size: 1336 bytes --] Instead of doing the wrapping in the smp code we can provide a default wrapper for those APICs which insist on cpumasks. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/include/asm/ipi.h | 1 + arch/x86/kernel/apic/ipi.c | 8 ++++++++ 2 files changed, 9 insertions(+) Index: linux/arch/x86/include/asm/ipi.h =================================================================== --- linux.orig/arch/x86/include/asm/ipi.h +++ linux/arch/x86/include/asm/ipi.h @@ -119,6 +119,7 @@ static inline void native_apic_mem_write(APIC_ICR, cfg); } +extern void default_send_IPI_single(int cpu, int vector); extern void default_send_IPI_single_phys(int cpu, int vector); extern void default_send_IPI_mask_sequence_phys(const struct cpumask *mask, int vector); Index: linux/arch/x86/kernel/apic/ipi.c =================================================================== --- linux.orig/arch/x86/kernel/apic/ipi.c +++ linux/arch/x86/kernel/apic/ipi.c @@ -65,6 +65,14 @@ void default_send_IPI_mask_allbutself_ph local_irq_restore(flags); } +/* + * Helper function for APICs which insist on cpumasks + */ +void default_send_IPI_single(int cpu, int vector) +{ + apic->send_IPI_mask(cpumask_of(cpu), vector); +} + #ifdef CONFIG_X86_32 void default_send_IPI_mask_sequence_logical(const struct cpumask *mask,
[-- Attachment #1: x86-apic--Use-default-send-single-IPI-wrapper --] [-- Type: text/plain, Size: 1319 bytes --] Wire up the default_send_IPI_single() wrapper to the last holdouts. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/apic_flat_64.c | 1 + arch/x86/kernel/apic/probe_32.c | 1 + 2 files changed, 2 insertions(+) Index: linux/arch/x86/kernel/apic/apic_flat_64.c =================================================================== --- linux.orig/arch/x86/kernel/apic/apic_flat_64.c +++ linux/arch/x86/kernel/apic/apic_flat_64.c @@ -185,6 +185,7 @@ static struct apic apic_flat = { .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and, + .send_IPI = default_send_IPI_single, .send_IPI_mask = flat_send_IPI_mask, .send_IPI_mask_allbutself = flat_send_IPI_mask_allbutself, .send_IPI_allbutself = flat_send_IPI_allbutself, Index: linux/arch/x86/kernel/apic/probe_32.c =================================================================== --- linux.orig/arch/x86/kernel/apic/probe_32.c +++ linux/arch/x86/kernel/apic/probe_32.c @@ -105,6 +105,7 @@ static struct apic apic_default = { .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and, + .send_IPI = default_send_IPI_single, .send_IPI_mask = default_send_IPI_mask_logical, .send_IPI_mask_allbutself = default_send_IPI_mask_allbutself_logical, .send_IPI_allbutself = default_send_IPI_allbutself,
[-- Attachment #1: x86-smp--Remove-single-IPI-wrapper --] [-- Type: text/plain, Size: 1390 bytes --] All APIC implementation have send_IPI now. Remove the conditional in the calling code. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/smp.c | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) Index: linux/arch/x86/kernel/smp.c =================================================================== --- linux.orig/arch/x86/kernel/smp.c +++ linux/arch/x86/kernel/smp.c @@ -115,18 +115,6 @@ static atomic_t stopping_cpu = ATOMIC_IN static bool smp_no_nmi_ipi = false; /* - * Helper wrapper: not all apic definitions support sending to - * a single CPU, so we fall back to sending to a mask. - */ -static void send_IPI_cpu(int cpu, int vector) -{ - if (apic->send_IPI) - apic->send_IPI(cpu, vector); - else - apic->send_IPI_mask(cpumask_of(cpu), vector); -} - -/* * this function sends a 'reschedule' IPI to another CPU. * it goes straight through and wastes no time serializing * anything. Worst case is that we lose a reschedule ... @@ -137,12 +125,12 @@ static void native_smp_send_reschedule(i WARN_ON(1); return; } - send_IPI_cpu(cpu, RESCHEDULE_VECTOR); + apic->send_IPI(cpu, RESCHEDULE_VECTOR); } void native_send_call_func_single_ipi(int cpu) { - send_IPI_cpu(cpu, CALL_FUNCTION_SINGLE_VECTOR); + apic->send_IPI(cpu, CALL_FUNCTION_SINGLE_VECTOR); } void native_send_call_func_ipi(const struct cpumask *mask)
On Wed, Nov 4, 2015 at 2:56 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
>
> Compiles and didn't explode when booting it, so it must be perfect :)
I have taught you well, young padawan.
Linus
* Thomas Gleixner <tglx@linutronix.de> wrote: > From: Linus Torvalds <torvalds@linux-foundation.org> > > [ tglx: Split it out from the patch which provides the new callback > and wrapped it into local_irq_save/restore ] > > Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> > Signed-off-by: Thomas Gleixner <tglx@linutronix.de> > --- > arch/x86/kernel/apic/x2apic_cluster.c | 12 ++++++++++++ > 1 file changed, 12 insertions(+) > > Index: linux/arch/x86/kernel/apic/x2apic_cluster.c > =================================================================== > --- linux.orig/arch/x86/kernel/apic/x2apic_cluster.c > +++ linux/arch/x86/kernel/apic/x2apic_cluster.c > @@ -23,6 +23,17 @@ static inline u32 x2apic_cluster(int cpu > return per_cpu(x86_cpu_to_logical_apicid, cpu) >> 16; > } > > +static void x2apic_send_IPI(int cpu, int vector) > +{ > + u32 dest = per_cpu(x86_cpu_to_logical_apicid, cpu); > + unsigned long flags; > + > + x2apic_wrmsr_fence(); > + local_irq_save(flags); > + __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); > + local_irq_restore(flags); > +} So the series looks good to me: Reviewed-by: Ingo Molnar <mingo@kernel.org> but in the above sequence I think we can do even better: we don't need the local_irq_save()/restore() I think. The reason, this is how __x2apic_send_IPI_dest() looks like: unsigned long cfg = __prepare_ICR(0, vector, dest); native_x2apic_icr_write(cfg, apicid); __prepare_ICR(), which is a confusing misnomer as it does not prepare anything about the ICR register, it just pre-calculates some values, is obviously interrupt-safe: static inline unsigned int __prepare_ICR(unsigned int shortcut, int vector, unsigned int dest) { unsigned int icr = shortcut | dest; switch (vector) { default: icr |= APIC_DM_FIXED | vector; break; case NMI_VECTOR: icr |= APIC_DM_NMI; break; } return icr; } and native_x2apic_icr_write() is a single WRMSR: static inline void native_x2apic_icr_write(u32 low, u32 id) { wrmsrl(APIC_BASE_MSR + (APIC_ICR >> 4), ((__u64) id) << 32 | low); } which is interrupt-safe as well. So we can save another 10-20 cycles of CLI/POPF overhead from this hotpath. I'd do it as a patch on top, to keep the series simpler - something like the below. (Completely untested: may the Force be with you.) Thanks, Ingo Signed-off-by: Ingo Molnar <mingo@kernel.org> --- arch/x86/kernel/apic/x2apic_cluster.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/x86/kernel/apic/x2apic_cluster.c b/arch/x86/kernel/apic/x2apic_cluster.c index 3329dab47efc..aca8b75c1552 100644 --- a/arch/x86/kernel/apic/x2apic_cluster.c +++ b/arch/x86/kernel/apic/x2apic_cluster.c @@ -26,12 +26,9 @@ static inline u32 x2apic_cluster(int cpu) static void x2apic_send_IPI(int cpu, int vector) { u32 dest = per_cpu(x86_cpu_to_logical_apicid, cpu); - unsigned long flags; x2apic_wrmsr_fence(); - local_irq_save(flags); __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); - local_irq_restore(flags); } static void
On Wed, Nov 4, 2015 at 10:38 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> but in the above sequence I think we can do even better: we don't need the
> local_irq_save()/restore() I think.
Right. Thomas added that one to my patch.
I don't think he realized just *how* trivial sending a single IPI is
with the x2apic, and just how horribly nasty the mask case is in
comparison (and how nasty it is that we historically turn a the single
ipi into a mask).
Linus
On Wed, 4 Nov 2015, Linus Torvalds wrote:
> On Wed, Nov 4, 2015 at 10:38 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > but in the above sequence I think we can do even better: we don't need the
> > local_irq_save()/restore() I think.
>
> Right. Thomas added that one to my patch.
>
> I don't think he realized just *how* trivial sending a single IPI is
> with the x2apic, and just how horribly nasty the mask case is in
> comparison (and how nasty it is that we historically turn a the single
> ipi into a mask).
Yeah, I was too tired to look at the details. Will fix that up for the
other trivial cases as well.
Thanks,
tglx
On Wed, 4 Nov 2015, Linus Torvalds wrote: > On Wed, Nov 4, 2015 at 2:56 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > > > > Compiles and didn't explode when booting it, so it must be perfect :) > > I have taught you well, young padawan. The young padawan, who is a grumpy old grandpa today, learned that from the old masters of SC/MP knights in the end of the 70s. Back then compilers were mainly operated in brain and aided by instruction cards, paper, pencils and rubber. It always compiled so the only way to be sure was running it, which was an extra tedious task on such beasts: http://www.computinghistory.org.uk/userdata/images/large/PRODPIC-4623.jpg Sorry, I had to disillusion you on that. tglx
Commit-ID: 539da7877275edb21a76aa02fb2c147eff02c559 Gitweb: http://git.kernel.org/tip/539da7877275edb21a76aa02fb2c147eff02c559 Author: Linus Torvalds <torvalds@linux-foundation.org> AuthorDate: Wed, 4 Nov 2015 22:57:00 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:51 +0100 x86/apic: Add a single-target IPI function to the apic We still fall back on the "send mask" versions if an apic definition doesn't have the single-target version, but at least this allows the (trivial) case for the common clustered x2apic case. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220848.737120838@linutronix.de Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/include/asm/apic.h | 1 + arch/x86/kernel/smp.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h index a30316b..7f62ad4 100644 --- a/arch/x86/include/asm/apic.h +++ b/arch/x86/include/asm/apic.h @@ -303,6 +303,7 @@ struct apic { unsigned int *apicid); /* ipi */ + void (*send_IPI)(int cpu, int vector); void (*send_IPI_mask)(const struct cpumask *mask, int vector); void (*send_IPI_mask_allbutself)(const struct cpumask *mask, int vector); diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c index 12c8286..1dbf590 100644 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -115,6 +115,18 @@ static atomic_t stopping_cpu = ATOMIC_INIT(-1); static bool smp_no_nmi_ipi = false; /* + * Helper wrapper: not all apic definitions support sending to + * a single CPU, so we fall back to sending to a mask. + */ +static void send_IPI_cpu(int cpu, int vector) +{ + if (apic->send_IPI) + apic->send_IPI(cpu, vector); + else + apic->send_IPI_mask(cpumask_of(cpu), vector); +} + +/* * this function sends a 'reschedule' IPI to another CPU. * it goes straight through and wastes no time serializing * anything. Worst case is that we lose a reschedule ... @@ -125,12 +137,12 @@ static void native_smp_send_reschedule(int cpu) WARN_ON(1); return; } - apic->send_IPI_mask(cpumask_of(cpu), RESCHEDULE_VECTOR); + send_IPI_cpu(cpu, RESCHEDULE_VECTOR); } void native_send_call_func_single_ipi(int cpu) { - apic->send_IPI_mask(cpumask_of(cpu), CALL_FUNCTION_SINGLE_VECTOR); + send_IPI_cpu(cpu, CALL_FUNCTION_SINGLE_VECTOR); } void native_send_call_func_ipi(const struct cpumask *mask)
Commit-ID: 7b6ce46cb3d096831dea3accacee4717c66abac8 Gitweb: http://git.kernel.org/tip/7b6ce46cb3d096831dea3accacee4717c66abac8 Author: Linus Torvalds <torvalds@linux-foundation.org> AuthorDate: Wed, 4 Nov 2015 22:57:00 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:52 +0100 x86/apic: Implement single target IPI function for x2apic_cluster [ tglx: Split it out from the patch which provides the new callback ] Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220848.817975597@linutronix.de Signed-off-by: Thomas Gleixner <tglx@linutronix.de> --- arch/x86/kernel/apic/x2apic_cluster.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kernel/apic/x2apic_cluster.c b/arch/x86/kernel/apic/x2apic_cluster.c index cc8311c..aca8b75 100644 --- a/arch/x86/kernel/apic/x2apic_cluster.c +++ b/arch/x86/kernel/apic/x2apic_cluster.c @@ -23,6 +23,14 @@ static inline u32 x2apic_cluster(int cpu) return per_cpu(x86_cpu_to_logical_apicid, cpu) >> 16; } +static void x2apic_send_IPI(int cpu, int vector) +{ + u32 dest = per_cpu(x86_cpu_to_logical_apicid, cpu); + + x2apic_wrmsr_fence(); + __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); +} + static void __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) { @@ -266,6 +274,7 @@ static struct apic apic_x2apic_cluster = { .cpu_mask_to_apicid_and = x2apic_cpu_mask_to_apicid_and, + .send_IPI = x2apic_send_IPI, .send_IPI_mask = x2apic_send_IPI_mask, .send_IPI_mask_allbutself = x2apic_send_IPI_mask_allbutself, .send_IPI_allbutself = x2apic_send_IPI_allbutself,
Commit-ID: 53be0fac8bdaeec87e0df7d0334345421d2be187 Gitweb: http://git.kernel.org/tip/53be0fac8bdaeec87e0df7d0334345421d2be187 Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:01 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:52 +0100 x86/apic: Implement default single target IPI function apic_physflat and bigsmp_apic can share that implementation. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220848.898543767@linutronix.de --- arch/x86/include/asm/ipi.h | 1 + arch/x86/kernel/apic/ipi.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/arch/x86/include/asm/ipi.h b/arch/x86/include/asm/ipi.h index 615fa90..22998a8 100644 --- a/arch/x86/include/asm/ipi.h +++ b/arch/x86/include/asm/ipi.h @@ -119,6 +119,7 @@ static inline void native_apic_mem_write(APIC_ICR, cfg); } +extern void default_send_IPI_single_phys(int cpu, int vector); extern void default_send_IPI_mask_sequence_phys(const struct cpumask *mask, int vector); extern void default_send_IPI_mask_allbutself_phys(const struct cpumask *mask, diff --git a/arch/x86/kernel/apic/ipi.c b/arch/x86/kernel/apic/ipi.c index 6207156..4fcffbf 100644 --- a/arch/x86/kernel/apic/ipi.c +++ b/arch/x86/kernel/apic/ipi.c @@ -18,6 +18,16 @@ #include <asm/proto.h> #include <asm/ipi.h> +void default_send_IPI_single_phys(int cpu, int vector) +{ + unsigned long flags; + + local_irq_save(flags); + __default_send_IPI_dest_field(per_cpu(x86_cpu_to_apicid, cpu), + vector, APIC_DEST_PHYSICAL); + local_irq_restore(flags); +} + void default_send_IPI_mask_sequence_phys(const struct cpumask *mask, int vector) { unsigned long query_cpu;
Commit-ID: 449112f4f35074f1dc70d4f0e769cb14150c159c Gitweb: http://git.kernel.org/tip/449112f4f35074f1dc70d4f0e769cb14150c159c Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:02 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:52 +0100 x86/apic: Remove pointless indirections from apic_physflat No value in having 32 byte extra text. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220848.975653382@linutronix.de --- arch/x86/kernel/apic/apic_flat_64.c | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c index f92ab36..6d3e1a6 100644 --- a/arch/x86/kernel/apic/apic_flat_64.c +++ b/arch/x86/kernel/apic/apic_flat_64.c @@ -230,17 +230,6 @@ static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id) return 0; } -static void physflat_send_IPI_mask(const struct cpumask *cpumask, int vector) -{ - default_send_IPI_mask_sequence_phys(cpumask, vector); -} - -static void physflat_send_IPI_mask_allbutself(const struct cpumask *cpumask, - int vector) -{ - default_send_IPI_mask_allbutself_phys(cpumask, vector); -} - static void physflat_send_IPI_allbutself(int vector) { default_send_IPI_mask_allbutself_phys(cpu_online_mask, vector); @@ -248,7 +237,7 @@ static void physflat_send_IPI_allbutself(int vector) static void physflat_send_IPI_all(int vector) { - physflat_send_IPI_mask(cpu_online_mask, vector); + default_send_IPI_mask_sequence_phys(cpu_online_mask, vector); } static int physflat_probe(void) @@ -292,8 +281,8 @@ static struct apic apic_physflat = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, - .send_IPI_mask = physflat_send_IPI_mask, - .send_IPI_mask_allbutself = physflat_send_IPI_mask_allbutself, + .send_IPI_mask = default_send_IPI_mask_sequence_phys, + .send_IPI_mask_allbutself = default_send_IPI_mask_allbutself_phys, .send_IPI_allbutself = physflat_send_IPI_allbutself, .send_IPI_all = physflat_send_IPI_all, .send_IPI_self = apic_send_IPI_self,
Commit-ID: 68cd88ff8df97846eb07080f17264a4de50cb012 Gitweb: http://git.kernel.org/tip/68cd88ff8df97846eb07080f17264a4de50cb012 Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:02 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:52 +0100 x86/apic: Wire up single IPI for apic_physflat Use the default implementation. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.055046864@linutronix.de --- arch/x86/kernel/apic/apic_flat_64.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c index 6d3e1a6..9de25d4 100644 --- a/arch/x86/kernel/apic/apic_flat_64.c +++ b/arch/x86/kernel/apic/apic_flat_64.c @@ -281,6 +281,7 @@ static struct apic apic_physflat = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = default_send_IPI_single_phys, .send_IPI_mask = default_send_IPI_mask_sequence_phys, .send_IPI_mask_allbutself = default_send_IPI_mask_allbutself_phys, .send_IPI_allbutself = physflat_send_IPI_allbutself,
Commit-ID: 500bd02fb17e5d9296c77ccc07db61fd5d4922a4 Gitweb: http://git.kernel.org/tip/500bd02fb17e5d9296c77ccc07db61fd5d4922a4 Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:03 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:52 +0100 x86/apic: Remove pointless indirections from bigsmp_apic Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.133086575@linutronix.de --- arch/x86/kernel/apic/bigsmp_32.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/apic/bigsmp_32.c b/arch/x86/kernel/apic/bigsmp_32.c index 971cf88..d4d103b 100644 --- a/arch/x86/kernel/apic/bigsmp_32.c +++ b/arch/x86/kernel/apic/bigsmp_32.c @@ -96,11 +96,6 @@ static int bigsmp_phys_pkg_id(int cpuid_apic, int index_msb) return cpuid_apic >> index_msb; } -static inline void bigsmp_send_IPI_mask(const struct cpumask *mask, int vector) -{ - default_send_IPI_mask_sequence_phys(mask, vector); -} - static void bigsmp_send_IPI_allbutself(int vector) { default_send_IPI_mask_allbutself_phys(cpu_online_mask, vector); @@ -108,7 +103,7 @@ static void bigsmp_send_IPI_allbutself(int vector) static void bigsmp_send_IPI_all(int vector) { - bigsmp_send_IPI_mask(cpu_online_mask, vector); + default_send_IPI_mask_sequence_phys(cpu_online_mask, vector); } static int dmi_bigsmp; /* can be set by dmi scanners */ @@ -180,7 +175,7 @@ static struct apic apic_bigsmp = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, - .send_IPI_mask = bigsmp_send_IPI_mask, + .send_IPI_mask = default_send_IPI_mask_sequence_phys, .send_IPI_mask_allbutself = NULL, .send_IPI_allbutself = bigsmp_send_IPI_allbutself, .send_IPI_all = bigsmp_send_IPI_all,
Commit-ID: 5789a12e28f7bf6a37564a5fc9ebc60dc86659b5 Gitweb: http://git.kernel.org/tip/5789a12e28f7bf6a37564a5fc9ebc60dc86659b5 Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:04 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:52 +0100 x86/apic: Wire up single IPI for bigsmp_apic Use the default implementation. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.213292642@linutronix.de --- arch/x86/kernel/apic/bigsmp_32.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/kernel/apic/bigsmp_32.c b/arch/x86/kernel/apic/bigsmp_32.c index d4d103b..cf9bd89 100644 --- a/arch/x86/kernel/apic/bigsmp_32.c +++ b/arch/x86/kernel/apic/bigsmp_32.c @@ -175,6 +175,7 @@ static struct apic apic_bigsmp = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = default_send_IPI_single_phys, .send_IPI_mask = default_send_IPI_mask_sequence_phys, .send_IPI_mask_allbutself = NULL, .send_IPI_allbutself = bigsmp_send_IPI_allbutself,
Commit-ID: f2bffe8a3eef42a1cd3393d56acd9fe598d2119c Gitweb: http://git.kernel.org/tip/f2bffe8a3eef42a1cd3393d56acd9fe598d2119c Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:04 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:53 +0100 x86/apic: Implement single IPI for x2apic_phys Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.296438009@linutronix.de --- arch/x86/kernel/apic/x2apic_phys.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kernel/apic/x2apic_phys.c b/arch/x86/kernel/apic/x2apic_phys.c index 662e915..a1242e2 100644 --- a/arch/x86/kernel/apic/x2apic_phys.c +++ b/arch/x86/kernel/apic/x2apic_phys.c @@ -36,6 +36,14 @@ static int x2apic_acpi_madt_oem_check(char *oem_id, char *oem_table_id) return x2apic_enabled() && (x2apic_phys || x2apic_fadt_phys()); } +static void x2apic_send_IPI(int cpu, int vector) +{ + u32 dest = per_cpu(x86_cpu_to_apicid, cpu); + + x2apic_wrmsr_fence(); + __x2apic_send_IPI_dest(dest, vector, APIC_DEST_PHYSICAL); +} + static void __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) { @@ -122,6 +130,7 @@ static struct apic apic_x2apic_phys = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = x2apic_send_IPI, .send_IPI_mask = x2apic_send_IPI_mask, .send_IPI_mask_allbutself = x2apic_send_IPI_mask_allbutself, .send_IPI_allbutself = x2apic_send_IPI_allbutself,
Commit-ID: 8642ea953d99fc037c1076e9a8b3a822025fb251 Gitweb: http://git.kernel.org/tip/8642ea953d99fc037c1076e9a8b3a822025fb251 Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:05 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:53 +0100 x86/apic: Wire up single IPI for x2apic_uv The function already exists. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.376775625@linutronix.de --- arch/x86/kernel/apic/x2apic_uv_x.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/kernel/apic/x2apic_uv_x.c b/arch/x86/kernel/apic/x2apic_uv_x.c index 4a13946..d760c6b 100644 --- a/arch/x86/kernel/apic/x2apic_uv_x.c +++ b/arch/x86/kernel/apic/x2apic_uv_x.c @@ -406,6 +406,7 @@ static struct apic __refdata apic_x2apic_uv_x = { .cpu_mask_to_apicid_and = uv_cpu_mask_to_apicid_and, + .send_IPI = uv_send_IPI_one, .send_IPI_mask = uv_send_IPI_mask, .send_IPI_mask_allbutself = uv_send_IPI_mask_allbutself, .send_IPI_allbutself = uv_send_IPI_allbutself,
Commit-ID: c61a0d31ba0ce75cb1b88bb4eb2f41a1b80bc90f Gitweb: http://git.kernel.org/tip/c61a0d31ba0ce75cb1b88bb4eb2f41a1b80bc90f Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:06 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:53 +0100 x86/apic: Wire up single IPI for apic_numachip The function already exists. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.551445489@linutronix.de --- arch/x86/kernel/apic/apic_numachip.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kernel/apic/apic_numachip.c b/arch/x86/kernel/apic/apic_numachip.c index 38dd5ef..69329a6 100644 --- a/arch/x86/kernel/apic/apic_numachip.c +++ b/arch/x86/kernel/apic/apic_numachip.c @@ -276,6 +276,7 @@ static const struct apic apic_numachip1 __refconst = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = numachip_send_IPI_one, .send_IPI_mask = numachip_send_IPI_mask, .send_IPI_mask_allbutself = numachip_send_IPI_mask_allbutself, .send_IPI_allbutself = numachip_send_IPI_allbutself, @@ -327,6 +328,7 @@ static const struct apic apic_numachip2 __refconst = { .cpu_mask_to_apicid_and = default_cpu_mask_to_apicid_and, + .send_IPI = numachip_send_IPI_one, .send_IPI_mask = numachip_send_IPI_mask, .send_IPI_mask_allbutself = numachip_send_IPI_mask_allbutself, .send_IPI_allbutself = numachip_send_IPI_allbutself,
Commit-ID: 4727da2eb1ec79fdc2acdd2f764b5b2aacab998c Gitweb: http://git.kernel.org/tip/4727da2eb1ec79fdc2acdd2f764b5b2aacab998c Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:06 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:53 +0100 x86/apic: Implement single IPI for apic_noop Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.455429817@linutronix.de --- arch/x86/kernel/apic/apic_noop.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kernel/apic/apic_noop.c b/arch/x86/kernel/apic/apic_noop.c index 0d96749..331a7a0 100644 --- a/arch/x86/kernel/apic/apic_noop.c +++ b/arch/x86/kernel/apic/apic_noop.c @@ -30,6 +30,7 @@ #include <asm/e820.h> static void noop_init_apic_ldr(void) { } +static void noop_send_IPI(int cpu, int vector) { } static void noop_send_IPI_mask(const struct cpumask *cpumask, int vector) { } static void noop_send_IPI_mask_allbutself(const struct cpumask *cpumask, int vector) { } static void noop_send_IPI_allbutself(int vector) { } @@ -144,6 +145,7 @@ struct apic apic_noop = { .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and, + .send_IPI = noop_send_IPI, .send_IPI_mask = noop_send_IPI_mask, .send_IPI_mask_allbutself = noop_send_IPI_mask_allbutself, .send_IPI_allbutself = noop_send_IPI_allbutself,
Commit-ID: 7e29393b20a1a863a5f9bf48dc71e5cff4035ff5 Gitweb: http://git.kernel.org/tip/7e29393b20a1a863a5f9bf48dc71e5cff4035ff5 Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:07 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:53 +0100 x86/apic: Provide default send single IPI wrapper Instead of doing the wrapping in the smp code we can provide a default wrapper for those APICs which insist on cpumasks. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.631111846@linutronix.de --- arch/x86/include/asm/ipi.h | 1 + arch/x86/kernel/apic/ipi.c | 8 ++++++++ 2 files changed, 9 insertions(+) diff --git a/arch/x86/include/asm/ipi.h b/arch/x86/include/asm/ipi.h index 22998a8..cfc9a0d 100644 --- a/arch/x86/include/asm/ipi.h +++ b/arch/x86/include/asm/ipi.h @@ -119,6 +119,7 @@ static inline void native_apic_mem_write(APIC_ICR, cfg); } +extern void default_send_IPI_single(int cpu, int vector); extern void default_send_IPI_single_phys(int cpu, int vector); extern void default_send_IPI_mask_sequence_phys(const struct cpumask *mask, int vector); diff --git a/arch/x86/kernel/apic/ipi.c b/arch/x86/kernel/apic/ipi.c index 4fcffbf..eb45fc9 100644 --- a/arch/x86/kernel/apic/ipi.c +++ b/arch/x86/kernel/apic/ipi.c @@ -65,6 +65,14 @@ void default_send_IPI_mask_allbutself_phys(const struct cpumask *mask, local_irq_restore(flags); } +/* + * Helper function for APICs which insist on cpumasks + */ +void default_send_IPI_single(int cpu, int vector) +{ + apic->send_IPI_mask(cpumask_of(cpu), vector); +} + #ifdef CONFIG_X86_32 void default_send_IPI_mask_sequence_logical(const struct cpumask *mask,
Commit-ID: 6153058a03f4cc5200b0b29e201caa11779ebca0 Gitweb: http://git.kernel.org/tip/6153058a03f4cc5200b0b29e201caa11779ebca0 Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:08 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:53 +0100 x86/apic: Use default send single IPI wrapper Wire up the default_send_IPI_single() wrapper to the last holdouts. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.711224890@linutronix.de --- arch/x86/kernel/apic/apic_flat_64.c | 1 + arch/x86/kernel/apic/probe_32.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c index 9de25d4..9968f30 100644 --- a/arch/x86/kernel/apic/apic_flat_64.c +++ b/arch/x86/kernel/apic/apic_flat_64.c @@ -185,6 +185,7 @@ static struct apic apic_flat = { .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and, + .send_IPI = default_send_IPI_single, .send_IPI_mask = flat_send_IPI_mask, .send_IPI_mask_allbutself = flat_send_IPI_mask_allbutself, .send_IPI_allbutself = flat_send_IPI_allbutself, diff --git a/arch/x86/kernel/apic/probe_32.c b/arch/x86/kernel/apic/probe_32.c index 7694ae6..f316e34 100644 --- a/arch/x86/kernel/apic/probe_32.c +++ b/arch/x86/kernel/apic/probe_32.c @@ -105,6 +105,7 @@ static struct apic apic_default = { .cpu_mask_to_apicid_and = flat_cpu_mask_to_apicid_and, + .send_IPI = default_send_IPI_single, .send_IPI_mask = default_send_IPI_mask_logical, .send_IPI_mask_allbutself = default_send_IPI_mask_allbutself_logical, .send_IPI_allbutself = default_send_IPI_allbutself,
Commit-ID: 72613184a1f076659e8a902d64351f50d3f9c990 Gitweb: http://git.kernel.org/tip/72613184a1f076659e8a902d64351f50d3f9c990 Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Wed, 4 Nov 2015 22:57:09 +0000 Committer: Thomas Gleixner <tglx@linutronix.de> CommitDate: Thu, 5 Nov 2015 13:07:54 +0100 x86/smp: Remove single IPI wrapper All APIC implementation have send_IPI now. Remove the conditional in the calling code. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Ingo Molnar <mingo@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Borislav Petkov <bp@alien.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Travis <travis@sgi.com> Cc: Daniel J Blueman <daniel@numascale.com> Link: http://lkml.kernel.org/r/20151104220849.807817097@linutronix.de --- arch/x86/kernel/smp.c | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c index 1dbf590..658777c 100644 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -115,18 +115,6 @@ static atomic_t stopping_cpu = ATOMIC_INIT(-1); static bool smp_no_nmi_ipi = false; /* - * Helper wrapper: not all apic definitions support sending to - * a single CPU, so we fall back to sending to a mask. - */ -static void send_IPI_cpu(int cpu, int vector) -{ - if (apic->send_IPI) - apic->send_IPI(cpu, vector); - else - apic->send_IPI_mask(cpumask_of(cpu), vector); -} - -/* * this function sends a 'reschedule' IPI to another CPU. * it goes straight through and wastes no time serializing * anything. Worst case is that we lose a reschedule ... @@ -137,12 +125,12 @@ static void native_smp_send_reschedule(int cpu) WARN_ON(1); return; } - send_IPI_cpu(cpu, RESCHEDULE_VECTOR); + apic->send_IPI(cpu, RESCHEDULE_VECTOR); } void native_send_call_func_single_ipi(int cpu) { - send_IPI_cpu(cpu, CALL_FUNCTION_SINGLE_VECTOR); + apic->send_IPI(cpu, CALL_FUNCTION_SINGLE_VECTOR); } void native_send_call_func_ipi(const struct cpumask *mask)