The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio"))

The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio"))

[chenxiang (M)]

Hi,

There are 128 cpus and 16 irqs for SAS controller in my system, and 
there are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, 
cpu32-63 for node1, cpu64-95 for node2, cpu96-127 for node3).
We use function pci_alloc_irq_vectors_affinity() to set the affinity of 
irqs.

I find that  before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: 
Spread vectors on node according to nr_cpu ratio")), the relationship 
between irqs and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15,
irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... 
irq15 bind to cpu120-127. But after the patch, the relationship is 
changed: irq0 bind to cpu32-39,
irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to 
cpu0-7, irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to 
cpu24-31.

I notice that before calling the sort() in function 
alloc_nodes_vectors(), the id of array node_vectors[] is from 0,1,2,3. 
But after function sort(), the index of array node_vectors[] is 1,2,3,0.
But i think it sorts according to the numbers of cpus in those nodes, so 
it should be the same as before calling sort() as the numbers of cpus in 
every node are 32.

Is it a bug of sort() or the usage of sort()?

Thanks,
Shawn

Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio"))

[Ming Lei]

On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote:
> Hi,
> 
> There are 128 cpus and 16 irqs for SAS controller in my system, and there
> are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for
> node1, cpu64-95 for node2, cpu96-127 for node3).
> We use function pci_alloc_irq_vectors_affinity() to set the affinity of
> irqs.
> 
> I find that  before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread
> vectors on node according to nr_cpu ratio")), the relationship between irqs
> and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15,
> irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15
> bind to cpu120-127. But after the patch, the relationship is changed: irq0
> bind to cpu32-39,
> irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7,
> irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31.
> 
> I notice that before calling the sort() in function alloc_nodes_vectors(),
> the id of array node_vectors[] is from 0,1,2,3. But after function sort(),
> the index of array node_vectors[] is 1,2,3,0.
> But i think it sorts according to the numbers of cpus in those nodes, so it
> should be the same as before calling sort() as the numbers of cpus in every
> node are 32.

Maybe there are more non-present CPUs covered by node 0.

Could you provide the following log?

1) lscpu

2) ./dump-io-irq-affinity $PCI_ID_SAS

	http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity

You need to figure out the PCI ID(the 1st column of lspci output) of the SAS
controller via lspci.

Thanks,
Ming

Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio"))

[Ming Lei]

[-- Attachment #1: Type: text/plain, Size: 2257 bytes --]

On Tue, Nov 19, 2019 at 11:05:55AM +0800, chenxiang (M) wrote:
> Hi Ming,
> 
> 在 2019/11/19 9:42, Ming Lei 写道:
> > On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote:
> > > Hi,
> > > 
> > > There are 128 cpus and 16 irqs for SAS controller in my system, and there
> > > are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for
> > > node1, cpu64-95 for node2, cpu96-127 for node3).
> > > We use function pci_alloc_irq_vectors_affinity() to set the affinity of
> > > irqs.
> > > 
> > > I find that  before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread
> > > vectors on node according to nr_cpu ratio")), the relationship between irqs
> > > and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15,
> > > irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15
> > > bind to cpu120-127. But after the patch, the relationship is changed: irq0
> > > bind to cpu32-39,
> > > irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7,
> > > irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31.
> > > 
> > > I notice that before calling the sort() in function alloc_nodes_vectors(),
> > > the id of array node_vectors[] is from 0,1,2,3. But after function sort(),
> > > the index of array node_vectors[] is 1,2,3,0.
> > > But i think it sorts according to the numbers of cpus in those nodes, so it
> > > should be the same as before calling sort() as the numbers of cpus in every
> > > node are 32.
> > Maybe there are more non-present CPUs covered by node 0.
> > 
> > Could you provide the following log?
> > 
> > 1) lscpu
> > 
> > 2) ./dump-io-irq-affinity $PCI_ID_SAS
> > 
> > 	http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity
> > 
> > You need to figure out the PCI ID(the 1st column of lspci output) of the SAS
> > controller via lspci.
> 
> Sorry, I can't access the link you provide, but i can provide those irqs'
> affinity in the attachment.
> I also write a small testcase, and find id is 1, 2, 3, 0 after calling
> sort() .

Runtime log from /proc/interrupts isn't useful for investigating
affinity allocation issue, please use the attached script for collecting
log.


Thanks,
Ming

[-- Attachment #2: dump-io-irq-affinity --]
[-- Type: text/plain, Size: 980 bytes --]

#!/bin/sh

get_disk_from_pcid()
{
	PCID=$1

	DISKS=`find /sys/block -name "*"`
	for DISK in $DISKS; do
		DISKP=`realpath $DISK/device`
		echo $DISKP | grep $PCID > /dev/null
		[ $? -eq 0 ] && echo `basename $DISK` && break
	done
}

dump_irq_affinity()
{
	PCID=$1
	PCIP=`find /sys/devices -name *$PCID | grep pci`

	[[ ! -d $PCIP/msi_irqs ]] && return

	IRQS=`ls $PCIP/msi_irqs`

	[ $? -ne 0 ] && return

	DISK=`get_disk_from_pcid $PCID`
	echo "PCI name is $PCID: $DISK"

	for IRQ in $IRQS; do
	    CPUS=`cat /proc/irq/$IRQ/smp_affinity_list`
	    ECPUS=`cat /proc/irq/$IRQ/effective_affinity_list`
	    echo -e "\tirq $IRQ, cpu list $CPUS, effective list $ECPUS"
	done
}


if [ $# -ge 1 ]; then
	PCIDS=$1
else
#	PCID=`lspci | grep "Non-Volatile memory" | cut -c1-7`
	PCIDS=`lspci | grep "Non-Volatile memory controller" | awk '{print $1}'`
fi

echo "kernel version: "
uname -a

for PCID in $PCIDS; do
	dump_irq_affinity $PCID
done

Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio"))

[chenxiang (M)]

在 2019/11/19 11:17, Ming Lei 写道:
> On Tue, Nov 19, 2019 at 11:05:55AM +0800, chenxiang (M) wrote:
>> Hi Ming,
>>
>> 在 2019/11/19 9:42, Ming Lei 写道:
>>> On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote:
>>>> Hi,
>>>>
>>>> There are 128 cpus and 16 irqs for SAS controller in my system, and there
>>>> are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for
>>>> node1, cpu64-95 for node2, cpu96-127 for node3).
>>>> We use function pci_alloc_irq_vectors_affinity() to set the affinity of
>>>> irqs.
>>>>
>>>> I find that  before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread
>>>> vectors on node according to nr_cpu ratio")), the relationship between irqs
>>>> and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15,
>>>> irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15
>>>> bind to cpu120-127. But after the patch, the relationship is changed: irq0
>>>> bind to cpu32-39,
>>>> irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7,
>>>> irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31.
>>>>
>>>> I notice that before calling the sort() in function alloc_nodes_vectors(),
>>>> the id of array node_vectors[] is from 0,1,2,3. But after function sort(),
>>>> the index of array node_vectors[] is 1,2,3,0.
>>>> But i think it sorts according to the numbers of cpus in those nodes, so it
>>>> should be the same as before calling sort() as the numbers of cpus in every
>>>> node are 32.
>>> Maybe there are more non-present CPUs covered by node 0.
>>>
>>> Could you provide the following log?
>>>
>>> 1) lscpu
>>>
>>> 2) ./dump-io-irq-affinity $PCI_ID_SAS
>>>
>>> 	http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity
>>>
>>> You need to figure out the PCI ID(the 1st column of lspci output) of the SAS
>>> controller via lspci.
>> Sorry, I can't access the link you provide, but i can provide those irqs'
>> affinity in the attachment.
>> I also write a small testcase, and find id is 1, 2, 3, 0 after calling
>> sort() .
> Runtime log from /proc/interrupts isn't useful for investigating
> affinity allocation issue, please use the attached script for collecting
> log.

Note: there are 32 irqs for SAS controller, irq0-15 are other interrupts 
(such as phy up/down/channel....), only irq 16-31 are cq interrupts 
which is processed by function pci_alloc_irq_vectors_affinity().
The log is as follows:

Euler:~ # ./dump-io-irq-affinity 74:02.0
kernel version:
Linux Euler 5.4.0-rc2-14683-g74684b1-dirty #224 SMP PREEMPT Mon Nov 18 
18:54:27 CST 2019 aarch64 aarch64 aarch64 GNU/Linux
PCI name is 74:02.0: sdd
cat: /proc/irq/65/smp_affinity_list: No such file or directory
cat: /proc/irq/65/effective_affinity_list: No such file or directory
     irq 65, cpu list , effective list
     irq 66, cpu list 0-31, effective list 0
     irq 67, cpu list 0-31, effective list 0
cat: /proc/irq/68/smp_affinity_list: No such file or directory
cat: /proc/irq/68/effective_affinity_list: No such file or directory
     irq 68, cpu list , effective list
cat: /proc/irq/69/smp_affinity_list: No such file or directory
cat: /proc/irq/69/effective_affinity_list: No such file or directory
     irq 69, cpu list , effective list
cat: /proc/irq/70/smp_affinity_list: No such file or directory
cat: /proc/irq/70/effective_affinity_list: No such file or directory
     irq 70, cpu list , effective list
cat: /proc/irq/71/smp_affinity_list: No such file or directory
cat: /proc/irq/71/effective_affinity_list: No such file or directory
     irq 71, cpu list , effective list
cat: /proc/irq/72/smp_affinity_list: No such file or directory
cat: /proc/irq/72/effective_affinity_list: No such file or directory
     irq 72, cpu list , effective list
cat: /proc/irq/73/smp_affinity_list: No such file or directory
cat: /proc/irq/73/effective_affinity_list: No such file or directory
     irq 73, cpu list , effective list
cat: /proc/irq/74/smp_affinity_list: No such file or directory
cat: /proc/irq/74/effective_affinity_list: No such file or directory
     irq 74, cpu list , effective list
cat: /proc/irq/75/smp_affinity_list: No such file or directory
cat: /proc/irq/75/effective_affinity_list: No such file or directory
     irq 75, cpu list , effective list
     irq 76, cpu list 0-31, effective list 0
cat: /proc/irq/77/smp_affinity_list: No such file or directory
cat: /proc/irq/77/effective_affinity_list: No such file or directory
     irq 77, cpu list , effective list
cat: /proc/irq/78/smp_affinity_list: No such file or directory
cat: /proc/irq/78/effective_affinity_list: No such file or directory
     irq 78, cpu list , effective list
cat: /proc/irq/79/smp_affinity_list: No such file or directory
cat: /proc/irq/79/effective_affinity_list: No such file or directory
     irq 79, cpu list , effective list
cat: /proc/irq/80/smp_affinity_list: No such file or directory
cat: /proc/irq/80/effective_affinity_list: No such file or directory
     irq 80, cpu list , effective list
     irq 81, cpu list 32-39, effective list 32
     irq 82, cpu list 40-47, effective list 40
     irq 83, cpu list 48-55, effective list 48
     irq 84, cpu list 56-63, effective list 56
     irq 85, cpu list 64-71, effective list 64
     irq 86, cpu list 72-79, effective list 72
     irq 87, cpu list 80-87, effective list 80
     irq 88, cpu list 88-95, effective list 88
     irq 89, cpu list 96-103, effective list 96
     irq 90, cpu list 104-111, effective list 104
     irq 91, cpu list 112-119, effective list 112
     irq 92, cpu list 120-127, effective list 120
     irq 93, cpu list 0-7, effective list 0
     irq 94, cpu list 8-15, effective list 8
     irq 95, cpu list 16-23, effective list 16
     irq 96, cpu list 24-31, effective list 24


>
>
> Thanks,
> Ming

Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio"))

[Ming Lei]

On Tue, Nov 19, 2019 at 11:32:48AM +0800, chenxiang (M) wrote:
> 
> 
> 在 2019/11/19 11:17, Ming Lei 写道:
> > On Tue, Nov 19, 2019 at 11:05:55AM +0800, chenxiang (M) wrote:
> > > Hi Ming,
> > > 
> > > 在 2019/11/19 9:42, Ming Lei 写道:
> > > > On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote:
> > > > > Hi,
> > > > > 
> > > > > There are 128 cpus and 16 irqs for SAS controller in my system, and there
> > > > > are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for
> > > > > node1, cpu64-95 for node2, cpu96-127 for node3).
> > > > > We use function pci_alloc_irq_vectors_affinity() to set the affinity of
> > > > > irqs.
> > > > > 
> > > > > I find that  before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread
> > > > > vectors on node according to nr_cpu ratio")), the relationship between irqs
> > > > > and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15,
> > > > > irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15
> > > > > bind to cpu120-127. But after the patch, the relationship is changed: irq0
> > > > > bind to cpu32-39,
> > > > > irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7,
> > > > > irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31.
> > > > > 
> > > > > I notice that before calling the sort() in function alloc_nodes_vectors(),
> > > > > the id of array node_vectors[] is from 0,1,2,3. But after function sort(),
> > > > > the index of array node_vectors[] is 1,2,3,0.
> > > > > But i think it sorts according to the numbers of cpus in those nodes, so it
> > > > > should be the same as before calling sort() as the numbers of cpus in every
> > > > > node are 32.
> > > > Maybe there are more non-present CPUs covered by node 0.
> > > > 
> > > > Could you provide the following log?
> > > > 
> > > > 1) lscpu
> > > > 
> > > > 2) ./dump-io-irq-affinity $PCI_ID_SAS
> > > > 
> > > > 	http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity
> > > > 
> > > > You need to figure out the PCI ID(the 1st column of lspci output) of the SAS
> > > > controller via lspci.
> > > Sorry, I can't access the link you provide, but i can provide those irqs'
> > > affinity in the attachment.
> > > I also write a small testcase, and find id is 1, 2, 3, 0 after calling
> > > sort() .
> > Runtime log from /proc/interrupts isn't useful for investigating
> > affinity allocation issue, please use the attached script for collecting
> > log.
> 
> Note: there are 32 irqs for SAS controller, irq0-15 are other interrupts
> (such as phy up/down/channel....), only irq 16-31 are cq interrupts which is
> processed by function pci_alloc_irq_vectors_affinity().
> The log is as follows:
> 
> Euler:~ # ./dump-io-irq-affinity 74:02.0
> kernel version:
> Linux Euler 5.4.0-rc2-14683-g74684b1-dirty #224 SMP PREEMPT Mon Nov 18
> 18:54:27 CST 2019 aarch64 aarch64 aarch64 GNU/Linux
> PCI name is 74:02.0: sdd
> cat: /proc/irq/65/smp_affinity_list: No such file or directory
> cat: /proc/irq/65/effective_affinity_list: No such file or directory
>     irq 65, cpu list , effective list
>     irq 66, cpu list 0-31, effective list 0
>     irq 67, cpu list 0-31, effective list 0
> cat: /proc/irq/68/smp_affinity_list: No such file or directory
> cat: /proc/irq/68/effective_affinity_list: No such file or directory
>     irq 68, cpu list , effective list
> cat: /proc/irq/69/smp_affinity_list: No such file or directory
> cat: /proc/irq/69/effective_affinity_list: No such file or directory
>     irq 69, cpu list , effective list
> cat: /proc/irq/70/smp_affinity_list: No such file or directory
> cat: /proc/irq/70/effective_affinity_list: No such file or directory
>     irq 70, cpu list , effective list
> cat: /proc/irq/71/smp_affinity_list: No such file or directory
> cat: /proc/irq/71/effective_affinity_list: No such file or directory
>     irq 71, cpu list , effective list
> cat: /proc/irq/72/smp_affinity_list: No such file or directory
> cat: /proc/irq/72/effective_affinity_list: No such file or directory
>     irq 72, cpu list , effective list
> cat: /proc/irq/73/smp_affinity_list: No such file or directory
> cat: /proc/irq/73/effective_affinity_list: No such file or directory
>     irq 73, cpu list , effective list
> cat: /proc/irq/74/smp_affinity_list: No such file or directory
> cat: /proc/irq/74/effective_affinity_list: No such file or directory
>     irq 74, cpu list , effective list
> cat: /proc/irq/75/smp_affinity_list: No such file or directory
> cat: /proc/irq/75/effective_affinity_list: No such file or directory
>     irq 75, cpu list , effective list
>     irq 76, cpu list 0-31, effective list 0
> cat: /proc/irq/77/smp_affinity_list: No such file or directory
> cat: /proc/irq/77/effective_affinity_list: No such file or directory
>     irq 77, cpu list , effective list
> cat: /proc/irq/78/smp_affinity_list: No such file or directory
> cat: /proc/irq/78/effective_affinity_list: No such file or directory
>     irq 78, cpu list , effective list
> cat: /proc/irq/79/smp_affinity_list: No such file or directory
> cat: /proc/irq/79/effective_affinity_list: No such file or directory
>     irq 79, cpu list , effective list
> cat: /proc/irq/80/smp_affinity_list: No such file or directory
> cat: /proc/irq/80/effective_affinity_list: No such file or directory
>     irq 80, cpu list , effective list
>     irq 81, cpu list 32-39, effective list 32
>     irq 82, cpu list 40-47, effective list 40
>     irq 83, cpu list 48-55, effective list 48
>     irq 84, cpu list 56-63, effective list 56
>     irq 85, cpu list 64-71, effective list 64
>     irq 86, cpu list 72-79, effective list 72
>     irq 87, cpu list 80-87, effective list 80
>     irq 88, cpu list 88-95, effective list 88
>     irq 89, cpu list 96-103, effective list 96
>     irq 90, cpu list 104-111, effective list 104
>     irq 91, cpu list 112-119, effective list 112
>     irq 92, cpu list 120-127, effective list 120
>     irq 93, cpu list 0-7, effective list 0
>     irq 94, cpu list 8-15, effective list 8
>     irq 95, cpu list 16-23, effective list 16
>     irq 96, cpu list 24-31, effective list 24

OK, nothing is wrong.

That is the implementation of sort().

If you provide the following array to sort():

	struct elem {
	        unsigned id;
	        unsigned val;
	};
	struct elem A[4] = {{0, 4}, {1, 4}, {2, 4}, {3, 4}};

And the output will becomes:

	A = {{1, 4}, {2, 4}, {3, 4}, {0, 4}};

It can't be problem for users of pci_alloc_irq_vectors_affinity() cause
all nodes are symmetric, can it?


Thanks,
Ming

Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio"))

[George Spelvin]

On Tue, 19 Nov 2019 at 11:05:55 +0800, chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> Sorry, I can't access the link you provide, but I can provide those
> irqs' affinity in the attachment.  I also write a small testcase,
> and find id is 1, 2, 3, 0 after calling sort().
> 
> struct ex_s {
>     int id;
>     int cpus;
> };
> struct ex_s ex_total[4] = {
>     {0, 8},
>     {1, 8},
>     {2, 8},
>     {3, 8}
> };
> 
> static int cmp_func(const void *l, const void *r)
> {
>     const struct ex_s *ln = l;
>     const struct ex_s *rn = r;
> 
>     return ln->cpus - rn->cpus;
> }

Your cmp_func is the problem.  sort() in the Linux kernel, like
qsort() in the C library, has never been stable, meaning that if
cmp_func() returns 0, there is no guarantee what order *l and *r
will end up in.  Minor changes to the implementation can change the
result.  You're sorting on the cpus field, which is all 8, so your
cmp_func is saying "I don't care what order the results appear in".

(A web search on "stable sort" will produce decades of discussion
on the subject, but basically an unstable sort has numerous
implementation advantages, and problems can usually be solved by
adjusting the comparison function.)

If you want to sort by ->id if ->cpus is the same, then change the
cmp_func to say so:

static int cmp_func(const void *l, const void *r)
{
    const struct ex_s *ln = l;
    const struct ex_s *rn = r;

    if (ln->cpus != rn->cpus)
	return ln->cpus - rn->cpus;
    else
	return ln->id - rn->id;
}
(This simple subtract-based compare depends on the fact that "cpus"
and "id" are both guaranteed to fit into 31 bits.  If there's any chance
The true difference could exceed INT_MAX, you need to get a bit fancier.)

Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio"))

[chenxiang (M)]

在 2019/12/8 15:42, George Spelvin 写道:
> On Tue, 19 Nov 2019 at 11:05:55 +0800, chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
>> Sorry, I can't access the link you provide, but I can provide those
>> irqs' affinity in the attachment.  I also write a small testcase,
>> and find id is 1, 2, 3, 0 after calling sort().
>>
>> struct ex_s {
>>      int id;
>>      int cpus;
>> };
>> struct ex_s ex_total[4] = {
>>      {0, 8},
>>      {1, 8},
>>      {2, 8},
>>      {3, 8}
>> };
>>
>> static int cmp_func(const void *l, const void *r)
>> {
>>      const struct ex_s *ln = l;
>>      const struct ex_s *rn = r;
>>
>>      return ln->cpus - rn->cpus;
>> }
> Your cmp_func is the problem.  sort() in the Linux kernel, like
> qsort() in the C library, has never been stable, meaning that if
> cmp_func() returns 0, there is no guarantee what order *l and *r
> will end up in.  Minor changes to the implementation can change the
> result.  You're sorting on the cpus field, which is all 8, so your
> cmp_func is saying "I don't care what order the results appear in".
>
> (A web search on "stable sort" will produce decades of discussion
> on the subject, but basically an unstable sort has numerous
> implementation advantages, and problems can usually be solved by
> adjusting the comparison function.)
>
> If you want to sort by ->id if ->cpus is the same, then change the
> cmp_func to say so:
>
> static int cmp_func(const void *l, const void *r)
> {
>      const struct ex_s *ln = l;
>      const struct ex_s *rn = r;
>
>      if (ln->cpus != rn->cpus)
> 	return ln->cpus - rn->cpus;
>      else
> 	return ln->id - rn->id;
> }
> (This simple subtract-based compare depends on the fact that "cpus"
> and "id" are both guaranteed to fit into 31 bits.  If there's any chance
> The true difference could exceed INT_MAX, you need to get a bit fancier.)
>
> .

Thank you for your reply,  George.
In function ncpus_cmp_func(), it only sorts by ->ncpus (which makes the 
affinity a little odd if the cpus of all the nodes are the same though 
it doesn't affect the function).
Maybe it is more better to sort by ->id if ->ncpus is the same in the 
function.