smp_affinity, MSI-X and 2.6.21.1

smp_affinity, MSI-X and 2.6.21.1

[Rick Jones]

Folks -

Is it a bug, or a feature that after changing a device's smp_affinity via echo 
"N" >> /proc/irq/M/smp_affinity that the new mask isn't visible via cat 
/proc/irq/M/smp_affinity until after actual interrupts are taken?

rick jones

Re: smp_affinity, MSI-X and 2.6.21.1

[Arjan van de Ven]

On Mon, 2007-05-07 at 13:53 -0700, Rick Jones wrote:
> Folks -
> 
> Is it a bug, or a feature that after changing a device's smp_affinity via echo 
> "N" >> /proc/irq/M/smp_affinity that the new mask isn't visible via cat 
> /proc/irq/M/smp_affinity until after actual interrupts are taken?\

that's known and expected behavior

Re: smp_affinity, MSI-X and 2.6.21.1

[Rick Jones]

Arjan van de Ven wrote:
> On Mon, 2007-05-07 at 13:53 -0700, Rick Jones wrote:
> 
>>Folks -
>>
>>Is it a bug, or a feature that after changing a device's smp_affinity via echo 
>>"N" >> /proc/irq/M/smp_affinity that the new mask isn't visible via cat 
>>/proc/irq/M/smp_affinity until after actual interrupts are taken?\
> 
> 
> that's known and expected behavior

well, expected by those in the know anyway :)

thanks.

rick jones

Re: smp_affinity, MSI-X and 2.6.21.1

[David Miller]

From: Andi Kleen <andi@firstfloor.org>
Date: 09 May 2007 12:35:29 +0200

> Rick Jones <rick.jones2@hp.com> writes:
> 
> > Folks -
> > 
> > Is it a bug, or a feature that after changing a device's smp_affinity
> > via echo "N" >> /proc/irq/M/smp_affinity that the new mask isn't
> > visible via cat /proc/irq/M/smp_affinity until after actual interrupts
> > are taken?
> 
> Intel chipsets can only safely update affinity during interrupt processing.
> You see a side effect of the code implementing this restriction.

That's true, but we are talking about software state so in some sense
it might be better that the affinity-to-be is reported to the user in
this case.

Delayed register updates are an implementation detail the user does
not need to know about here.

Re: smp_affinity, MSI-X and 2.6.21.1

[Andi Kleen]

Rick Jones <rick.jones2@hp.com> writes:

> Folks -
> 
> Is it a bug, or a feature that after changing a device's smp_affinity
> via echo "N" >> /proc/irq/M/smp_affinity that the new mask isn't
> visible via cat /proc/irq/M/smp_affinity until after actual interrupts
> are taken?

Intel chipsets can only safely update affinity during interrupt processing.
You see a side effect of the code implementing this restriction.

-Andi

Re: smp_affinity, MSI-X and 2.6.21.1

[Andi Kleen]

> That's true, but we are talking about software state so in some sense
> it might be better that the affinity-to-be is reported to the user in
> this case.
> 
> Delayed register updates are an implementation detail the user does
> not need to know about here.

This patch should fix it.
-Andi

Report the pending irq if available in smp_affinity

Otherwise smp_affinity would only update after the next interrupt
on x86 systems.

Cc: tglx@linutronix.de
Cc: mingo@elte.hu

Signed-off-by: Andi Kleen <ak@suse.de>

Index: linux/kernel/irq/proc.c
===================================================================
--- linux.orig/kernel/irq/proc.c
+++ linux/kernel/irq/proc.c
@@ -19,7 +19,14 @@ static struct proc_dir_entry *root_irq_d
 static int irq_affinity_read_proc(char *page, char **start, off_t off,
 				  int count, int *eof, void *data)
 {
-	int len = cpumask_scnprintf(page, count, irq_desc[(long)data].affinity);
+	struct irq_desc *desc = irq_desc + (long)data;
+	cpumask_t *mask = &desc->affinity;
+	int len;
+#ifdef CONFIG_GENERIC_PENDING_IRQ
+	if (desc->status & IRQ_MOVE_PENDING)
+		mask = &desc->pending_mask;
+#endif
+	len = cpumask_scnprintf(page, count, *mask);
 
 	if (count - len < 2)
 		return -EINVAL;

Re: smp_affinity, MSI-X and 2.6.21.1

[Rick Jones]

Andi Kleen wrote:
>>That's true, but we are talking about software state so in some sense
>>it might be better that the affinity-to-be is reported to the user in
>>this case.
>>
>>Delayed register updates are an implementation detail the user does
>>not need to know about here.
> 
> 
> This patch should fix it.

And it seems to when I apply it against the 2.6.21.1 kernel I'm messing about with:

hpcpc106:~/s2io-2.0.19-8893# cat /proc/irq/69/smp_affinity
ffffffff,ffffffff
hpcpc106:~/s2io-2.0.19-8893# echo "4" >> /proc/irq/69/smp_affinity
hpcpc106:~/s2io-2.0.19-8893# cat /proc/irq/69/smp_affinity
00000000,00000004
hpcpc106:~/s2io-2.0.19-8893# cat /proc/interrupts | grep 69
  69:          0          0          0          0         PCI-MSI  eth2:MSI-X-6-RX

It would be nice if this could find its way into the kernel at some point - 
2.6.23 or 2.6.24 perhaps?

rick jones

> -Andi
> 
> Report the pending irq if available in smp_affinity
> 
> Otherwise smp_affinity would only update after the next interrupt
> on x86 systems.
> 
> Cc: tglx@linutronix.de
> Cc: mingo@elte.hu
> 
> Signed-off-by: Andi Kleen <ak@suse.de>
> 
> Index: linux/kernel/irq/proc.c
> ===================================================================
> --- linux.orig/kernel/irq/proc.c
> +++ linux/kernel/irq/proc.c
> @@ -19,7 +19,14 @@ static struct proc_dir_entry *root_irq_d
>  static int irq_affinity_read_proc(char *page, char **start, off_t off,
>  				  int count, int *eof, void *data)
>  {
> -	int len = cpumask_scnprintf(page, count, irq_desc[(long)data].affinity);
> +	struct irq_desc *desc = irq_desc + (long)data;
> +	cpumask_t *mask = &desc->affinity;
> +	int len;
> +#ifdef CONFIG_GENERIC_PENDING_IRQ
> +	if (desc->status & IRQ_MOVE_PENDING)
> +		mask = &desc->pending_mask;
> +#endif
> +	len = cpumask_scnprintf(page, count, *mask);
>  
>  	if (count - len < 2)
>  		return -EINVAL;