* The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) @ 2019-11-19 1:25 chenxiang (M) 2019-11-19 1:42 ` Ming Lei 0 siblings, 1 reply; 7+ messages in thread From: chenxiang (M) @ 2019-11-19 1:25 UTC (permalink / raw) To: Ming Lei, lkml, tglx; +Cc: kbusch, axboe, linux-block, Linuxarm, John Garry Hi, There are 128 cpus and 16 irqs for SAS controller in my system, and there are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for node1, cpu64-95 for node2, cpu96-127 for node3). We use function pci_alloc_irq_vectors_affinity() to set the affinity of irqs. I find that before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")), the relationship between irqs and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15, irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15 bind to cpu120-127. But after the patch, the relationship is changed: irq0 bind to cpu32-39, irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7, irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31. I notice that before calling the sort() in function alloc_nodes_vectors(), the id of array node_vectors[] is from 0,1,2,3. But after function sort(), the index of array node_vectors[] is 1,2,3,0. But i think it sorts according to the numbers of cpus in those nodes, so it should be the same as before calling sort() as the numbers of cpus in every node are 32. Is it a bug of sort() or the usage of sort()? Thanks, Shawn ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) 2019-11-19 1:25 The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) chenxiang (M) @ 2019-11-19 1:42 ` Ming Lei [not found] ` <a8a89884-8323-ff70-f35e-0fcf5d7afefc@hisilicon.com> 0 siblings, 1 reply; 7+ messages in thread From: Ming Lei @ 2019-11-19 1:42 UTC (permalink / raw) To: chenxiang (M) Cc: lkml, tglx, kbusch, axboe, linux-block, Linuxarm, John Garry On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote: > Hi, > > There are 128 cpus and 16 irqs for SAS controller in my system, and there > are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for > node1, cpu64-95 for node2, cpu96-127 for node3). > We use function pci_alloc_irq_vectors_affinity() to set the affinity of > irqs. > > I find that before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread > vectors on node according to nr_cpu ratio")), the relationship between irqs > and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15, > irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15 > bind to cpu120-127. But after the patch, the relationship is changed: irq0 > bind to cpu32-39, > irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7, > irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31. > > I notice that before calling the sort() in function alloc_nodes_vectors(), > the id of array node_vectors[] is from 0,1,2,3. But after function sort(), > the index of array node_vectors[] is 1,2,3,0. > But i think it sorts according to the numbers of cpus in those nodes, so it > should be the same as before calling sort() as the numbers of cpus in every > node are 32. Maybe there are more non-present CPUs covered by node 0. Could you provide the following log? 1) lscpu 2) ./dump-io-irq-affinity $PCI_ID_SAS http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity You need to figure out the PCI ID(the 1st column of lspci output) of the SAS controller via lspci. Thanks, Ming ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <a8a89884-8323-ff70-f35e-0fcf5d7afefc@hisilicon.com>]
* Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) [not found] ` <a8a89884-8323-ff70-f35e-0fcf5d7afefc@hisilicon.com> @ 2019-11-19 3:17 ` Ming Lei 2019-11-19 3:32 ` chenxiang (M) 2019-12-08 7:42 ` George Spelvin 1 sibling, 1 reply; 7+ messages in thread From: Ming Lei @ 2019-11-19 3:17 UTC (permalink / raw) To: chenxiang (M) Cc: lkml, tglx, kbusch, axboe, linux-block, Linuxarm, John Garry [-- Attachment #1: Type: text/plain, Size: 2257 bytes --] On Tue, Nov 19, 2019 at 11:05:55AM +0800, chenxiang (M) wrote: > Hi Ming, > > 在 2019/11/19 9:42, Ming Lei 写道: > > On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote: > > > Hi, > > > > > > There are 128 cpus and 16 irqs for SAS controller in my system, and there > > > are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for > > > node1, cpu64-95 for node2, cpu96-127 for node3). > > > We use function pci_alloc_irq_vectors_affinity() to set the affinity of > > > irqs. > > > > > > I find that before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread > > > vectors on node according to nr_cpu ratio")), the relationship between irqs > > > and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15, > > > irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15 > > > bind to cpu120-127. But after the patch, the relationship is changed: irq0 > > > bind to cpu32-39, > > > irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7, > > > irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31. > > > > > > I notice that before calling the sort() in function alloc_nodes_vectors(), > > > the id of array node_vectors[] is from 0,1,2,3. But after function sort(), > > > the index of array node_vectors[] is 1,2,3,0. > > > But i think it sorts according to the numbers of cpus in those nodes, so it > > > should be the same as before calling sort() as the numbers of cpus in every > > > node are 32. > > Maybe there are more non-present CPUs covered by node 0. > > > > Could you provide the following log? > > > > 1) lscpu > > > > 2) ./dump-io-irq-affinity $PCI_ID_SAS > > > > http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity > > > > You need to figure out the PCI ID(the 1st column of lspci output) of the SAS > > controller via lspci. > > Sorry, I can't access the link you provide, but i can provide those irqs' > affinity in the attachment. > I also write a small testcase, and find id is 1, 2, 3, 0 after calling > sort() . Runtime log from /proc/interrupts isn't useful for investigating affinity allocation issue, please use the attached script for collecting log. Thanks, Ming [-- Attachment #2: dump-io-irq-affinity --] [-- Type: text/plain, Size: 980 bytes --] #!/bin/sh get_disk_from_pcid() { PCID=$1 DISKS=`find /sys/block -name "*"` for DISK in $DISKS; do DISKP=`realpath $DISK/device` echo $DISKP | grep $PCID > /dev/null [ $? -eq 0 ] && echo `basename $DISK` && break done } dump_irq_affinity() { PCID=$1 PCIP=`find /sys/devices -name *$PCID | grep pci` [[ ! -d $PCIP/msi_irqs ]] && return IRQS=`ls $PCIP/msi_irqs` [ $? -ne 0 ] && return DISK=`get_disk_from_pcid $PCID` echo "PCI name is $PCID: $DISK" for IRQ in $IRQS; do CPUS=`cat /proc/irq/$IRQ/smp_affinity_list` ECPUS=`cat /proc/irq/$IRQ/effective_affinity_list` echo -e "\tirq $IRQ, cpu list $CPUS, effective list $ECPUS" done } if [ $# -ge 1 ]; then PCIDS=$1 else # PCID=`lspci | grep "Non-Volatile memory" | cut -c1-7` PCIDS=`lspci | grep "Non-Volatile memory controller" | awk '{print $1}'` fi echo "kernel version: " uname -a for PCID in $PCIDS; do dump_irq_affinity $PCID done ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) 2019-11-19 3:17 ` Ming Lei @ 2019-11-19 3:32 ` chenxiang (M) 2019-11-19 6:56 ` Ming Lei 0 siblings, 1 reply; 7+ messages in thread From: chenxiang (M) @ 2019-11-19 3:32 UTC (permalink / raw) To: Ming Lei; +Cc: lkml, tglx, kbusch, axboe, linux-block, Linuxarm, John Garry 在 2019/11/19 11:17, Ming Lei 写道: > On Tue, Nov 19, 2019 at 11:05:55AM +0800, chenxiang (M) wrote: >> Hi Ming, >> >> 在 2019/11/19 9:42, Ming Lei 写道: >>> On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote: >>>> Hi, >>>> >>>> There are 128 cpus and 16 irqs for SAS controller in my system, and there >>>> are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for >>>> node1, cpu64-95 for node2, cpu96-127 for node3). >>>> We use function pci_alloc_irq_vectors_affinity() to set the affinity of >>>> irqs. >>>> >>>> I find that before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread >>>> vectors on node according to nr_cpu ratio")), the relationship between irqs >>>> and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15, >>>> irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15 >>>> bind to cpu120-127. But after the patch, the relationship is changed: irq0 >>>> bind to cpu32-39, >>>> irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7, >>>> irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31. >>>> >>>> I notice that before calling the sort() in function alloc_nodes_vectors(), >>>> the id of array node_vectors[] is from 0,1,2,3. But after function sort(), >>>> the index of array node_vectors[] is 1,2,3,0. >>>> But i think it sorts according to the numbers of cpus in those nodes, so it >>>> should be the same as before calling sort() as the numbers of cpus in every >>>> node are 32. >>> Maybe there are more non-present CPUs covered by node 0. >>> >>> Could you provide the following log? >>> >>> 1) lscpu >>> >>> 2) ./dump-io-irq-affinity $PCI_ID_SAS >>> >>> http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity >>> >>> You need to figure out the PCI ID(the 1st column of lspci output) of the SAS >>> controller via lspci. >> Sorry, I can't access the link you provide, but i can provide those irqs' >> affinity in the attachment. >> I also write a small testcase, and find id is 1, 2, 3, 0 after calling >> sort() . > Runtime log from /proc/interrupts isn't useful for investigating > affinity allocation issue, please use the attached script for collecting > log. Note: there are 32 irqs for SAS controller, irq0-15 are other interrupts (such as phy up/down/channel....), only irq 16-31 are cq interrupts which is processed by function pci_alloc_irq_vectors_affinity(). The log is as follows: Euler:~ # ./dump-io-irq-affinity 74:02.0 kernel version: Linux Euler 5.4.0-rc2-14683-g74684b1-dirty #224 SMP PREEMPT Mon Nov 18 18:54:27 CST 2019 aarch64 aarch64 aarch64 GNU/Linux PCI name is 74:02.0: sdd cat: /proc/irq/65/smp_affinity_list: No such file or directory cat: /proc/irq/65/effective_affinity_list: No such file or directory irq 65, cpu list , effective list irq 66, cpu list 0-31, effective list 0 irq 67, cpu list 0-31, effective list 0 cat: /proc/irq/68/smp_affinity_list: No such file or directory cat: /proc/irq/68/effective_affinity_list: No such file or directory irq 68, cpu list , effective list cat: /proc/irq/69/smp_affinity_list: No such file or directory cat: /proc/irq/69/effective_affinity_list: No such file or directory irq 69, cpu list , effective list cat: /proc/irq/70/smp_affinity_list: No such file or directory cat: /proc/irq/70/effective_affinity_list: No such file or directory irq 70, cpu list , effective list cat: /proc/irq/71/smp_affinity_list: No such file or directory cat: /proc/irq/71/effective_affinity_list: No such file or directory irq 71, cpu list , effective list cat: /proc/irq/72/smp_affinity_list: No such file or directory cat: /proc/irq/72/effective_affinity_list: No such file or directory irq 72, cpu list , effective list cat: /proc/irq/73/smp_affinity_list: No such file or directory cat: /proc/irq/73/effective_affinity_list: No such file or directory irq 73, cpu list , effective list cat: /proc/irq/74/smp_affinity_list: No such file or directory cat: /proc/irq/74/effective_affinity_list: No such file or directory irq 74, cpu list , effective list cat: /proc/irq/75/smp_affinity_list: No such file or directory cat: /proc/irq/75/effective_affinity_list: No such file or directory irq 75, cpu list , effective list irq 76, cpu list 0-31, effective list 0 cat: /proc/irq/77/smp_affinity_list: No such file or directory cat: /proc/irq/77/effective_affinity_list: No such file or directory irq 77, cpu list , effective list cat: /proc/irq/78/smp_affinity_list: No such file or directory cat: /proc/irq/78/effective_affinity_list: No such file or directory irq 78, cpu list , effective list cat: /proc/irq/79/smp_affinity_list: No such file or directory cat: /proc/irq/79/effective_affinity_list: No such file or directory irq 79, cpu list , effective list cat: /proc/irq/80/smp_affinity_list: No such file or directory cat: /proc/irq/80/effective_affinity_list: No such file or directory irq 80, cpu list , effective list irq 81, cpu list 32-39, effective list 32 irq 82, cpu list 40-47, effective list 40 irq 83, cpu list 48-55, effective list 48 irq 84, cpu list 56-63, effective list 56 irq 85, cpu list 64-71, effective list 64 irq 86, cpu list 72-79, effective list 72 irq 87, cpu list 80-87, effective list 80 irq 88, cpu list 88-95, effective list 88 irq 89, cpu list 96-103, effective list 96 irq 90, cpu list 104-111, effective list 104 irq 91, cpu list 112-119, effective list 112 irq 92, cpu list 120-127, effective list 120 irq 93, cpu list 0-7, effective list 0 irq 94, cpu list 8-15, effective list 8 irq 95, cpu list 16-23, effective list 16 irq 96, cpu list 24-31, effective list 24 > > > Thanks, > Ming ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) 2019-11-19 3:32 ` chenxiang (M) @ 2019-11-19 6:56 ` Ming Lei 0 siblings, 0 replies; 7+ messages in thread From: Ming Lei @ 2019-11-19 6:56 UTC (permalink / raw) To: chenxiang (M) Cc: lkml, tglx, kbusch, axboe, linux-block, Linuxarm, John Garry On Tue, Nov 19, 2019 at 11:32:48AM +0800, chenxiang (M) wrote: > > > 在 2019/11/19 11:17, Ming Lei 写道: > > On Tue, Nov 19, 2019 at 11:05:55AM +0800, chenxiang (M) wrote: > > > Hi Ming, > > > > > > 在 2019/11/19 9:42, Ming Lei 写道: > > > > On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote: > > > > > Hi, > > > > > > > > > > There are 128 cpus and 16 irqs for SAS controller in my system, and there > > > > > are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for > > > > > node1, cpu64-95 for node2, cpu96-127 for node3). > > > > > We use function pci_alloc_irq_vectors_affinity() to set the affinity of > > > > > irqs. > > > > > > > > > > I find that before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread > > > > > vectors on node according to nr_cpu ratio")), the relationship between irqs > > > > > and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15, > > > > > irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15 > > > > > bind to cpu120-127. But after the patch, the relationship is changed: irq0 > > > > > bind to cpu32-39, > > > > > irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7, > > > > > irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31. > > > > > > > > > > I notice that before calling the sort() in function alloc_nodes_vectors(), > > > > > the id of array node_vectors[] is from 0,1,2,3. But after function sort(), > > > > > the index of array node_vectors[] is 1,2,3,0. > > > > > But i think it sorts according to the numbers of cpus in those nodes, so it > > > > > should be the same as before calling sort() as the numbers of cpus in every > > > > > node are 32. > > > > Maybe there are more non-present CPUs covered by node 0. > > > > > > > > Could you provide the following log? > > > > > > > > 1) lscpu > > > > > > > > 2) ./dump-io-irq-affinity $PCI_ID_SAS > > > > > > > > http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity > > > > > > > > You need to figure out the PCI ID(the 1st column of lspci output) of the SAS > > > > controller via lspci. > > > Sorry, I can't access the link you provide, but i can provide those irqs' > > > affinity in the attachment. > > > I also write a small testcase, and find id is 1, 2, 3, 0 after calling > > > sort() . > > Runtime log from /proc/interrupts isn't useful for investigating > > affinity allocation issue, please use the attached script for collecting > > log. > > Note: there are 32 irqs for SAS controller, irq0-15 are other interrupts > (such as phy up/down/channel....), only irq 16-31 are cq interrupts which is > processed by function pci_alloc_irq_vectors_affinity(). > The log is as follows: > > Euler:~ # ./dump-io-irq-affinity 74:02.0 > kernel version: > Linux Euler 5.4.0-rc2-14683-g74684b1-dirty #224 SMP PREEMPT Mon Nov 18 > 18:54:27 CST 2019 aarch64 aarch64 aarch64 GNU/Linux > PCI name is 74:02.0: sdd > cat: /proc/irq/65/smp_affinity_list: No such file or directory > cat: /proc/irq/65/effective_affinity_list: No such file or directory > irq 65, cpu list , effective list > irq 66, cpu list 0-31, effective list 0 > irq 67, cpu list 0-31, effective list 0 > cat: /proc/irq/68/smp_affinity_list: No such file or directory > cat: /proc/irq/68/effective_affinity_list: No such file or directory > irq 68, cpu list , effective list > cat: /proc/irq/69/smp_affinity_list: No such file or directory > cat: /proc/irq/69/effective_affinity_list: No such file or directory > irq 69, cpu list , effective list > cat: /proc/irq/70/smp_affinity_list: No such file or directory > cat: /proc/irq/70/effective_affinity_list: No such file or directory > irq 70, cpu list , effective list > cat: /proc/irq/71/smp_affinity_list: No such file or directory > cat: /proc/irq/71/effective_affinity_list: No such file or directory > irq 71, cpu list , effective list > cat: /proc/irq/72/smp_affinity_list: No such file or directory > cat: /proc/irq/72/effective_affinity_list: No such file or directory > irq 72, cpu list , effective list > cat: /proc/irq/73/smp_affinity_list: No such file or directory > cat: /proc/irq/73/effective_affinity_list: No such file or directory > irq 73, cpu list , effective list > cat: /proc/irq/74/smp_affinity_list: No such file or directory > cat: /proc/irq/74/effective_affinity_list: No such file or directory > irq 74, cpu list , effective list > cat: /proc/irq/75/smp_affinity_list: No such file or directory > cat: /proc/irq/75/effective_affinity_list: No such file or directory > irq 75, cpu list , effective list > irq 76, cpu list 0-31, effective list 0 > cat: /proc/irq/77/smp_affinity_list: No such file or directory > cat: /proc/irq/77/effective_affinity_list: No such file or directory > irq 77, cpu list , effective list > cat: /proc/irq/78/smp_affinity_list: No such file or directory > cat: /proc/irq/78/effective_affinity_list: No such file or directory > irq 78, cpu list , effective list > cat: /proc/irq/79/smp_affinity_list: No such file or directory > cat: /proc/irq/79/effective_affinity_list: No such file or directory > irq 79, cpu list , effective list > cat: /proc/irq/80/smp_affinity_list: No such file or directory > cat: /proc/irq/80/effective_affinity_list: No such file or directory > irq 80, cpu list , effective list > irq 81, cpu list 32-39, effective list 32 > irq 82, cpu list 40-47, effective list 40 > irq 83, cpu list 48-55, effective list 48 > irq 84, cpu list 56-63, effective list 56 > irq 85, cpu list 64-71, effective list 64 > irq 86, cpu list 72-79, effective list 72 > irq 87, cpu list 80-87, effective list 80 > irq 88, cpu list 88-95, effective list 88 > irq 89, cpu list 96-103, effective list 96 > irq 90, cpu list 104-111, effective list 104 > irq 91, cpu list 112-119, effective list 112 > irq 92, cpu list 120-127, effective list 120 > irq 93, cpu list 0-7, effective list 0 > irq 94, cpu list 8-15, effective list 8 > irq 95, cpu list 16-23, effective list 16 > irq 96, cpu list 24-31, effective list 24 OK, nothing is wrong. That is the implementation of sort(). If you provide the following array to sort(): struct elem { unsigned id; unsigned val; }; struct elem A[4] = {{0, 4}, {1, 4}, {2, 4}, {3, 4}}; And the output will becomes: A = {{1, 4}, {2, 4}, {3, 4}, {0, 4}}; It can't be problem for users of pci_alloc_irq_vectors_affinity() cause all nodes are symmetric, can it? Thanks, Ming ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) [not found] ` <a8a89884-8323-ff70-f35e-0fcf5d7afefc@hisilicon.com> 2019-11-19 3:17 ` Ming Lei @ 2019-12-08 7:42 ` George Spelvin 2019-12-09 2:58 ` chenxiang (M) 1 sibling, 1 reply; 7+ messages in thread From: George Spelvin @ 2019-12-08 7:42 UTC (permalink / raw) To: chenxiang66, ming.lei Cc: axboe, john.garry, kbusch, linux-block, linuxarm, lkml, tglx On Tue, 19 Nov 2019 at 11:05:55 +0800, chenxiang (M)" <chenxiang66@hisilicon.com> wrote: > Sorry, I can't access the link you provide, but I can provide those > irqs' affinity in the attachment. I also write a small testcase, > and find id is 1, 2, 3, 0 after calling sort(). > > struct ex_s { > int id; > int cpus; > }; > struct ex_s ex_total[4] = { > {0, 8}, > {1, 8}, > {2, 8}, > {3, 8} > }; > > static int cmp_func(const void *l, const void *r) > { > const struct ex_s *ln = l; > const struct ex_s *rn = r; > > return ln->cpus - rn->cpus; > } Your cmp_func is the problem. sort() in the Linux kernel, like qsort() in the C library, has never been stable, meaning that if cmp_func() returns 0, there is no guarantee what order *l and *r will end up in. Minor changes to the implementation can change the result. You're sorting on the cpus field, which is all 8, so your cmp_func is saying "I don't care what order the results appear in". (A web search on "stable sort" will produce decades of discussion on the subject, but basically an unstable sort has numerous implementation advantages, and problems can usually be solved by adjusting the comparison function.) If you want to sort by ->id if ->cpus is the same, then change the cmp_func to say so: static int cmp_func(const void *l, const void *r) { const struct ex_s *ln = l; const struct ex_s *rn = r; if (ln->cpus != rn->cpus) return ln->cpus - rn->cpus; else return ln->id - rn->id; } (This simple subtract-based compare depends on the fact that "cpus" and "id" are both guaranteed to fit into 31 bits. If there's any chance The true difference could exceed INT_MAX, you need to get a bit fancier.) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) 2019-12-08 7:42 ` George Spelvin @ 2019-12-09 2:58 ` chenxiang (M) 0 siblings, 0 replies; 7+ messages in thread From: chenxiang (M) @ 2019-12-09 2:58 UTC (permalink / raw) To: George Spelvin, ming.lei Cc: axboe, john.garry, kbusch, linux-block, linuxarm, tglx 在 2019/12/8 15:42, George Spelvin 写道: > On Tue, 19 Nov 2019 at 11:05:55 +0800, chenxiang (M)" <chenxiang66@hisilicon.com> wrote: >> Sorry, I can't access the link you provide, but I can provide those >> irqs' affinity in the attachment. I also write a small testcase, >> and find id is 1, 2, 3, 0 after calling sort(). >> >> struct ex_s { >> int id; >> int cpus; >> }; >> struct ex_s ex_total[4] = { >> {0, 8}, >> {1, 8}, >> {2, 8}, >> {3, 8} >> }; >> >> static int cmp_func(const void *l, const void *r) >> { >> const struct ex_s *ln = l; >> const struct ex_s *rn = r; >> >> return ln->cpus - rn->cpus; >> } > Your cmp_func is the problem. sort() in the Linux kernel, like > qsort() in the C library, has never been stable, meaning that if > cmp_func() returns 0, there is no guarantee what order *l and *r > will end up in. Minor changes to the implementation can change the > result. You're sorting on the cpus field, which is all 8, so your > cmp_func is saying "I don't care what order the results appear in". > > (A web search on "stable sort" will produce decades of discussion > on the subject, but basically an unstable sort has numerous > implementation advantages, and problems can usually be solved by > adjusting the comparison function.) > > If you want to sort by ->id if ->cpus is the same, then change the > cmp_func to say so: > > static int cmp_func(const void *l, const void *r) > { > const struct ex_s *ln = l; > const struct ex_s *rn = r; > > if (ln->cpus != rn->cpus) > return ln->cpus - rn->cpus; > else > return ln->id - rn->id; > } > (This simple subtract-based compare depends on the fact that "cpus" > and "id" are both guaranteed to fit into 31 bits. If there's any chance > The true difference could exceed INT_MAX, you need to get a bit fancier.) > > . Thank you for your reply, George. In function ncpus_cmp_func(), it only sorts by ->ncpus (which makes the affinity a little odd if the cpus of all the nodes are the same though it doesn't affect the function). Maybe it is more better to sort by ->id if ->ncpus is the same in the function. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-12-09 2:58 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-11-19 1:25 The irq Affinity is changed after the patch(Fixes: b1a5a73e64e9 ("genirq/affinity: Spread vectors on node according to nr_cpu ratio")) chenxiang (M) 2019-11-19 1:42 ` Ming Lei [not found] ` <a8a89884-8323-ff70-f35e-0fcf5d7afefc@hisilicon.com> 2019-11-19 3:17 ` Ming Lei 2019-11-19 3:32 ` chenxiang (M) 2019-11-19 6:56 ` Ming Lei 2019-12-08 7:42 ` George Spelvin 2019-12-09 2:58 ` chenxiang (M)
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.