[PATCH v3] x86, irq: get correct available vectors for cpu disable

[PATCH v3] x86, irq: get correct available vectors for cpu disable

[Yinghai Lu]

used_vectors is a bitmap for vectors that are not tracked in per_cpu
vector_irq.
used_vectors contains information on the first 32 exceptions, the system vectors.
the IA32_SYSCALL_VECTOR (0x80), and the IRQ_MOVE_CLEANUP_VECTOR (0x20).

assign_irq_vectors() assigns vectors up to first_system_vector and
it will not use vectors that are set used_vectors.

This patch modifies the code to scan up to first_system_vector
and do a test on the used_vectors bitmap.

So count avaiable vectors correctly.

-v2: fix compiling problem.
-v3: update changelog and commets

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/kernel/irq.c |   18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

Index: linux-2.6/arch/x86/kernel/irq.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/irq.c
+++ linux-2.6/arch/x86/kernel/irq.c
@@ -17,6 +17,7 @@
 #include <asm/idle.h>
 #include <asm/mce.h>
 #include <asm/hw_irq.h>
+#include <asm/desc.h>
 
 #define CREATE_TRACE_POINTS
 #include <asm/trace/irq_vectors.h>
@@ -321,8 +322,21 @@ int check_irq_vectors_for_cpu_disable(vo
 	for_each_online_cpu(cpu) {
 		if (cpu == this_cpu)
 			continue;
-		for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS;
-		     vector++) {
+
+		/*
+		 * assign_irq_vector() only scan per_cpu vectors from
+		 * FIRST_EXTERNAL_VECTOR to first_system_vector.
+		 * It aslo skip vectors that are set in used_vectors bitmask.
+		 * used_vectors could have bits set for
+		 *	IA32_SYSCALL_VECTOR (0x80)
+		 *	IRQ_MOVE_CLEANUP_VECTOR (0x20)
+		 * Don't count those as available vectors.
+		 */
+		for (vector = FIRST_EXTERNAL_VECTOR;
+		     vector < first_system_vector; vector++) {
+			if (test_bit(vector, used_vectors))
+				continue;
+
 			if (per_cpu(vector_irq, cpu)[vector] < 0)
 				count++;
 		}

Re: [PATCH v3] x86, irq: get correct available vectors for cpu disable

[Prarit Bhargava]

On 01/28/2014 04:54 PM, Yinghai Lu wrote:
> used_vectors is a bitmap for vectors that are not tracked in per_cpu
> vector_irq.
> used_vectors contains information on the first 32 exceptions, the system vectors.
> the IA32_SYSCALL_VECTOR (0x80), and the IRQ_MOVE_CLEANUP_VECTOR (0x20).
> 
> assign_irq_vectors() assigns vectors up to first_system_vector and
> it will not use vectors that are set used_vectors.
> 
> This patch modifies the code to scan up to first_system_vector
> and do a test on the used_vectors bitmap.
> 
> So count avaiable vectors correctly.
> 
> -v2: fix compiling problem.
> -v3: update changelog and commets
> 

Yinghai, I have not forgotten about this patch.  I'm waiting for the system to
become available so that I can test it.  My apologies for the inconvenience,

P.

> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> 
> ---
>  arch/x86/kernel/irq.c |   18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/arch/x86/kernel/irq.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/irq.c
> +++ linux-2.6/arch/x86/kernel/irq.c
> @@ -17,6 +17,7 @@
>  #include <asm/idle.h>
>  #include <asm/mce.h>
>  #include <asm/hw_irq.h>
> +#include <asm/desc.h>
>  
>  #define CREATE_TRACE_POINTS
>  #include <asm/trace/irq_vectors.h>
> @@ -321,8 +322,21 @@ int check_irq_vectors_for_cpu_disable(vo
>  	for_each_online_cpu(cpu) {
>  		if (cpu == this_cpu)
>  			continue;
> -		for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS;
> -		     vector++) {
> +
> +		/*
> +		 * assign_irq_vector() only scan per_cpu vectors from
> +		 * FIRST_EXTERNAL_VECTOR to first_system_vector.
> +		 * It aslo skip vectors that are set in used_vectors bitmask.
> +		 * used_vectors could have bits set for
> +		 *	IA32_SYSCALL_VECTOR (0x80)
> +		 *	IRQ_MOVE_CLEANUP_VECTOR (0x20)
> +		 * Don't count those as available vectors.
> +		 */
> +		for (vector = FIRST_EXTERNAL_VECTOR;
> +		     vector < first_system_vector; vector++) {
> +			if (test_bit(vector, used_vectors))
> +				continue;
> +
>  			if (per_cpu(vector_irq, cpu)[vector] < 0)
>  				count++;
>  		}

Re: [PATCH v3] x86, irq: get correct available vectors for cpu disable

[Prarit Bhargava]

On 01/28/2014 04:54 PM, Yinghai Lu wrote:
> used_vectors is a bitmap for vectors that are not tracked in per_cpu
> vector_irq.
> used_vectors contains information on the first 32 exceptions, the system vectors.
> the IA32_SYSCALL_VECTOR (0x80), and the IRQ_MOVE_CLEANUP_VECTOR (0x20).
> 
> assign_irq_vectors() assigns vectors up to first_system_vector and
> it will not use vectors that are set used_vectors.
> 
> This patch modifies the code to scan up to first_system_vector
> and do a test on the used_vectors bitmap.
> 
> So count avaiable vectors correctly.
> 
> -v2: fix compiling problem.
> -v3: update changelog and commets
> 
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> 
> ---
>  arch/x86/kernel/irq.c |   18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/arch/x86/kernel/irq.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/irq.c
> +++ linux-2.6/arch/x86/kernel/irq.c
> @@ -17,6 +17,7 @@
>  #include <asm/idle.h>
>  #include <asm/mce.h>
>  #include <asm/hw_irq.h>
> +#include <asm/desc.h>
>  
>  #define CREATE_TRACE_POINTS
>  #include <asm/trace/irq_vectors.h>
> @@ -321,8 +322,21 @@ int check_irq_vectors_for_cpu_disable(vo
>  	for_each_online_cpu(cpu) {
>  		if (cpu == this_cpu)
>  			continue;
> -		for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS;
> -		     vector++) {
> +
> +		/*
> +		 * assign_irq_vector() only scan per_cpu vectors from
> +		 * FIRST_EXTERNAL_VECTOR to first_system_vector.
> +		 * It aslo skip vectors that are set in used_vectors bitmask.

"also" ... but not a big deal IMO.

Reviewed-by: Prarit Bhargava <prarit@redhat.com>
Tested-by: Prarit Bhargava <prarit@redhat.com>

P.

> +		 * used_vectors could have bits set for
> +		 *	IA32_SYSCALL_VECTOR (0x80)
> +		 *	IRQ_MOVE_CLEANUP_VECTOR (0x20)
> +		 * Don't count those as available vectors.
> +		 */
> +		for (vector = FIRST_EXTERNAL_VECTOR;
> +		     vector < first_system_vector; vector++) {
> +			if (test_bit(vector, used_vectors))
> +				continue;
> +
>  			if (per_cpu(vector_irq, cpu)[vector] < 0)
>  				count++;
>  		}

Re: [PATCH v3] x86, irq: get correct available vectors for cpu disable

[Linn Crosetto]

Thanks for the patch.

On Tue, Jan 28, 2014 at 01:54:05PM -0800, Yinghai Lu wrote:
> used_vectors is a bitmap for vectors that are not tracked in per_cpu
> vector_irq.

I feel like this comment (also in the code) could be misleading because vectors
above first_system_vector are effectively not tracked in per_cpu vector_irq, but
also may not have the bit set in used_vectors. For example, used_vectors from a
system that I am looking at now:

first_system_vector
                239                  255
                  |                  |
                  10 01000 11111 11111

test_bit(240, used_vectors) does not return the correct answer to the question
about whether the vector is tracked in per_cpu vector_irq. This leads to two
meanings for the bitmap; for vectors less than first_system_vector whether or
not they are tracked in per_cpu vector_irq, and for vectors above
first_system_vector, whether or not they are in use:

static inline int is_per_cpu_vector(int vector) {
	return !test_bit(vector, used_vectors) &&
			vector < first_system_vector;
}

Thanks,
Linn

> used_vectors contains information on the first 32 exceptions, the system vectors.
> the IA32_SYSCALL_VECTOR (0x80), and the IRQ_MOVE_CLEANUP_VECTOR (0x20).
> 
> assign_irq_vectors() assigns vectors up to first_system_vector and
> it will not use vectors that are set used_vectors.
> 
> This patch modifies the code to scan up to first_system_vector
> and do a test on the used_vectors bitmap.
> 
> So count avaiable vectors correctly.
> 
> -v2: fix compiling problem.
> -v3: update changelog and commets
> 
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> 
> ---
>  arch/x86/kernel/irq.c |   18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/arch/x86/kernel/irq.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/irq.c
> +++ linux-2.6/arch/x86/kernel/irq.c
> @@ -17,6 +17,7 @@
>  #include <asm/idle.h>
>  #include <asm/mce.h>
>  #include <asm/hw_irq.h>
> +#include <asm/desc.h>
>  
>  #define CREATE_TRACE_POINTS
>  #include <asm/trace/irq_vectors.h>
> @@ -321,8 +322,21 @@ int check_irq_vectors_for_cpu_disable(vo
>  	for_each_online_cpu(cpu) {
>  		if (cpu == this_cpu)
>  			continue;
> -		for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS;
> -		     vector++) {
> +
> +		/*
> +		 * assign_irq_vector() only scan per_cpu vectors from
> +		 * FIRST_EXTERNAL_VECTOR to first_system_vector.
> +		 * It aslo skip vectors that are set in used_vectors bitmask.
> +		 * used_vectors could have bits set for
> +		 *	IA32_SYSCALL_VECTOR (0x80)
> +		 *	IRQ_MOVE_CLEANUP_VECTOR (0x20)
> +		 * Don't count those as available vectors.
> +		 */
> +		for (vector = FIRST_EXTERNAL_VECTOR;
> +		     vector < first_system_vector; vector++) {
> +			if (test_bit(vector, used_vectors))
> +				continue;
> +
>  			if (per_cpu(vector_irq, cpu)[vector] < 0)
>  				count++;
>  		}

Re: [PATCH v3] x86, irq: get correct available vectors for cpu disable

[Yinghai Lu]

On Tue, Mar 25, 2014 at 1:03 PM, Linn Crosetto <linn@hp.com> wrote:
> Thanks for the patch.
>
> On Tue, Jan 28, 2014 at 01:54:05PM -0800, Yinghai Lu wrote:
>> used_vectors is a bitmap for vectors that are not tracked in per_cpu
>> vector_irq.
>
> I feel like this comment (also in the code) could be misleading because vectors
> above first_system_vector are effectively not tracked in per_cpu vector_irq, but
> also may not have the bit set in used_vectors. For example, used_vectors from a
> system that I am looking at now:
>
> first_system_vector
>                 239                  255
>                   |                  |
>                   10 01000 11111 11111
>
> test_bit(240, used_vectors) does not return the correct answer to the question
> about whether the vector is tracked in per_cpu vector_irq. This leads to two
> meanings for the bitmap; for vectors less than first_system_vector whether or
> not they are tracked in per_cpu vector_irq, and for vectors above
> first_system_vector, whether or not they are in use:
>
> static inline int is_per_cpu_vector(int vector) {
>         return !test_bit(vector, used_vectors) &&
>                         vector < first_system_vector;
> }

sorry, I can not catch what you want to say.

Do you mean the change log or comment in the patch is not right?

Thanks

Yinghai

Re: [PATCH v3] x86, irq: get correct available vectors for cpu disable

[Linn Crosetto]

On Tue, Mar 25, 2014 at 05:18:53PM -0700, Yinghai Lu wrote:
> On Tue, Mar 25, 2014 at 1:03 PM, Linn Crosetto <linn@hp.com> wrote:
> > Thanks for the patch.
> >
> > On Tue, Jan 28, 2014 at 01:54:05PM -0800, Yinghai Lu wrote:
> >> used_vectors is a bitmap for vectors that are not tracked in per_cpu
> >> vector_irq.
> >
> > I feel like this comment (also in the code) could be misleading because vectors
> > above first_system_vector are effectively not tracked in per_cpu vector_irq, but
> > also may not have the bit set in used_vectors. For example, used_vectors from a
> > system that I am looking at now:
> >
> > first_system_vector
> >                 239                  255
> >                   |                  |
> >                   10 01000 11111 11111
> >
> > test_bit(240, used_vectors) does not return the correct answer to the question
> > about whether the vector is tracked in per_cpu vector_irq. This leads to two
> > meanings for the bitmap; for vectors less than first_system_vector whether or
> > not they are tracked in per_cpu vector_irq, and for vectors above
> > first_system_vector, whether or not they are in use:
> >
> > static inline int is_per_cpu_vector(int vector) {
> >         return !test_bit(vector, used_vectors) &&
> >                         vector < first_system_vector;
> > }
> 
> sorry, I can not catch what you want to say.
> 
> Do you mean the change log or comment in the patch is not right?

Just noting that not all bits above first_system_vector are set in the bitmap,
so the comment in asm/desc.h and the change log could be misleading:

/* used_vectors is BITMAP for irq is not managed by percpu vector_irq */

I have tested and the patch itself is good.

Thanks,
Linn

Re: [PATCH v3] x86, irq: get correct available vectors for cpu disable

[Yinghai Lu]

On Wed, Mar 26, 2014 at 9:50 AM, Linn Crosetto <linn@hp.com> wrote:
> On Tue, Mar 25, 2014 at 05:18:53PM -0700, Yinghai Lu wrote:
>
> Just noting that not all bits above first_system_vector are set in the bitmap,
> so the comment in asm/desc.h and the change log could be misleading:
>
> /* used_vectors is BITMAP for irq is not managed by percpu vector_irq */
>

Do you mean some vector that is neither used by percpu_vector_irq nor
system_vector
after first_system_vector ?

Anyway if bit is set in used_vectors, that vector can not be used by
percpu vector_irq.

so that statement looks still right.

Feel free to suggest right comment or changelog.

> I have tested and the patch itself is good.

Good.

Thanks

Yinghai

Re: [PATCH v3] x86, irq: get correct available vectors for cpu disable

[Linn Crosetto]

On Wed, Mar 26, 2014 at 10:04:52AM -0700, Yinghai Lu wrote:
> On Wed, Mar 26, 2014 at 9:50 AM, Linn Crosetto <linn@hp.com> wrote:
> > On Tue, Mar 25, 2014 at 05:18:53PM -0700, Yinghai Lu wrote:
> >
> > Just noting that not all bits above first_system_vector are set in the bitmap,
> > so the comment in asm/desc.h and the change log could be misleading:
> >
> > /* used_vectors is BITMAP for irq is not managed by percpu vector_irq */
> >
> 
> Do you mean some vector that is neither used by percpu_vector_irq nor
> system_vector
> after first_system_vector ?

Yes, that is the case. In my previous example, first_system_vector is 239, but
test_bit(240, used_vectors) would return unset, though it cannot be used by
percpu vector_irq. If it marked all vectors unusable by percpu vector_irq, then
all bits above first_system_vector would be set.

> Anyway if bit is set in used_vectors, that vector can not be used by
> percpu vector_irq.
> 
> so that statement looks still right.
> 
> Feel free to suggest right comment or changelog.

The following might be more accurate:

/* 
 * The used_vectors BITMAP marks IRQs not managed by percpu vector_irq, below
 * first_system_vector. Vectors above first_system_vector are not managed by
 * percpu vector_irq.
 */