[patch 1/3] generic-ipi: add smp_mb() before sending the IPI

[patch 1/3] generic-ipi: add smp_mb() before sending the IPI

[Suresh Siddha]

[-- Attachment #1: generic_ipi_strengthen_barrier.patch --]
[-- Type: text/plain, Size: 1207 bytes --]

smp_mb() (to make the memory operations visible globally) is needed before
sending the ipi, if the receiver refers to the data(setup by the sender) in
a lock-free fashion.

On x86, x2apic mode accesses for sending IPI's don't have serializing
semantics. So the need for smp_mb() before sending the IPI becomes more
critical in x2apic mode.

Signed-off-by: Suresh Siddha <suresh.b.siddha@intel.com>
---

Index: linux-2.6.git/kernel/smp.c
===================================================================
--- linux-2.6.git.orig/kernel/smp.c	2008-10-31 11:15:11.000000000 -0700
+++ linux-2.6.git/kernel/smp.c	2008-11-04 09:54:29.000000000 -0800
@@ -76,6 +76,11 @@
 	list_add_tail(&data->list, &dst->list);
 	spin_unlock_irqrestore(&dst->lock, flags);
 
+	/*
+	 * Make the list addition visible before sending the ipi.
+	 */
+	smp_mb();
+
 	if (ipi)
 		arch_send_call_function_single_ipi(cpu);
 
@@ -370,6 +375,11 @@
 	list_add_tail_rcu(&data->csd.list, &call_function_queue);
 	spin_unlock_irqrestore(&call_function_lock, flags);
 
+	/*
+	 * Make the list addition visible before sending the ipi.
+	 */
+	smp_mb();
+
 	/* Send a message to all CPUs in the map */
 	arch_send_call_function_ipi(mask);
 

-- 

[patch 2/3] x86: Add smp_mb() before sending INVALIDATE_TLB_VECTOR

[Suresh Siddha]

[-- Attachment #1: x86_fix_mb_tlb_flush.patch --]
[-- Type: text/plain, Size: 1468 bytes --]

On x86, x2apic mode accesses for sending IPI's don't have serializing
semantics. If the IPI receiver refers(in lock-free fashion) to some memory setup
by the sender, the need for smp_mb() before sending the IPI becomes critical in
x2apic mode.

Add the smp_mb() in native_flush_tlb_others() before sending the IPI.

Signed-off-by: Suresh Siddha <suresh.b.siddha@intel.com>
---

Index: linux-2.6.git/arch/x86/kernel/tlb_32.c
===================================================================
--- linux-2.6.git.orig/arch/x86/kernel/tlb_32.c	2008-10-13 10:04:12.000000000 -0700
+++ linux-2.6.git/arch/x86/kernel/tlb_32.c	2008-11-04 10:01:16.000000000 -0800
@@ -154,6 +154,12 @@
 	flush_mm = mm;
 	flush_va = va;
 	cpus_or(flush_cpumask, cpumask, flush_cpumask);
+
+	/*
+	 * Make the above memory operations globally visible before
+	 * sending the IPI.
+	 */
+	smp_mb();
 	/*
 	 * We have to send the IPI only to
 	 * CPUs affected.
Index: linux-2.6.git/arch/x86/kernel/tlb_64.c
===================================================================
--- linux-2.6.git.orig/arch/x86/kernel/tlb_64.c	2008-10-31 11:15:11.000000000 -0700
+++ linux-2.6.git/arch/x86/kernel/tlb_64.c	2008-11-04 10:01:31.000000000 -0800
@@ -183,6 +183,11 @@
 	cpus_or(f->flush_cpumask, cpumask, f->flush_cpumask);
 
 	/*
+	 * Make the above memory operations globally visible before
+	 * sending the IPI.
+	 */
+	smp_mb();
+	/*
 	 * We have to send the IPI only to
 	 * CPUs affected.
 	 */

-- 

[patch 3/3] generic-ipi: fix the smp_mb() usage

[Suresh Siddha]

[-- Attachment #1: generic_ipi_weaken_barrier.patch --]
[-- Type: text/plain, Size: 1331 bytes --]

Remove the unnecessary smp_mb() in csd_flag_wait(), as the presence of that
smp_mb() doesn't mean anything on the sender, when the ipi receiver is not
doing any thing special (like memory fence) after clearing the CSD_FLAG_WAIT.

IPI receiver (on Alpha atleast) needs smp_read_barrier_depends() in the handler
before reading the call_single_queue list in a lock-free fashion. Replace
the existing smp_mb() (which is unnecessary) with smp_read_barrier_depends()

Signed-off-by: Suresh Siddha <suresh.b.siddha@intel.com>
---

Index: linux-2.6.git/kernel/smp.c
===================================================================
--- linux-2.6.git.orig/kernel/smp.c	2008-11-04 09:54:29.000000000 -0800
+++ linux-2.6.git/kernel/smp.c	2008-11-04 10:04:54.000000000 -0800
@@ -51,10 +51,6 @@
 {
 	/* Wait for response */
 	do {
-		/*
-		 * We need to see the flags store in the IPI handler
-		 */
-		smp_mb();
 		if (!(data->flags & CSD_FLAG_WAIT))
 			break;
 		cpu_relax();
@@ -162,7 +158,7 @@
 	 * Need to see other stores to list head for checking whether
 	 * list is empty without holding q->lock
 	 */
-	smp_mb();
+	smp_read_barrier_depends();
 	while (!list_empty(&q->list)) {
 		unsigned int data_flags;
 
@@ -196,7 +192,7 @@
 		/*
 		 * See comment on outer loop
 		 */
-		smp_mb();
+		smp_read_barrier_depends();
 	}
 }
 

-- 

Re: [patch 2/3] x86: Add smp_mb() before sending INVALIDATE_TLB_VECTOR

[Ingo Molnar]

* Suresh Siddha <suresh.b.siddha@intel.com> wrote:

> On x86, x2apic mode accesses for sending IPI's don't have 
> serializing semantics. If the IPI receiver refers(in lock-free 
> fashion) to some memory setup by the sender, the need for smp_mb() 
> before sending the IPI becomes critical in x2apic mode.
> 
> Add the smp_mb() in native_flush_tlb_others() before sending the 
> IPI.
> 
> Signed-off-by: Suresh Siddha <suresh.b.siddha@intel.com>

applied to tip/x86/urgent, thanks Suresh!

Jens, you'll send 1/3 and 3/3 to Linus, right?

	Ingo

Re: [patch 2/3] x86: Add smp_mb() before sending INVALIDATE_TLB_VECTOR

[Jens Axboe]

On Thu, Nov 06 2008, Ingo Molnar wrote:
> 
> * Suresh Siddha <suresh.b.siddha@intel.com> wrote:
> 
> > On x86, x2apic mode accesses for sending IPI's don't have 
> > serializing semantics. If the IPI receiver refers(in lock-free 
> > fashion) to some memory setup by the sender, the need for smp_mb() 
> > before sending the IPI becomes critical in x2apic mode.
> > 
> > Add the smp_mb() in native_flush_tlb_others() before sending the 
> > IPI.
> > 
> > Signed-off-by: Suresh Siddha <suresh.b.siddha@intel.com>
> 
> applied to tip/x86/urgent, thanks Suresh!
> 
> Jens, you'll send 1/3 and 3/3 to Linus, right?

Yes, it's in for-linus and I'll send the pull request today.

-- 
Jens Axboe