* [PATCH] RDMA/srp: Fix support for unpopulated and unbalanced NUMA nodes
@ 2021-03-17 7:45 Yi Zhang
2021-03-19 9:49 ` Greg KH
0 siblings, 1 reply; 4+ messages in thread
From: Yi Zhang @ 2021-03-17 7:45 UTC (permalink / raw)
To: stable; +Cc: jgg, Nicolas Morey-Chaisemartin, Bart Van Assche
From: Nicolas Morey-Chaisemartin <nmoreychaisemartin@suse.com>
This patch fixed one kernel NULL pointer issue with blktests srp/005
----------
From: Nicolas Morey-Chaisemartin <nmoreychaisemartin@suse.com>
commit 2b5715fc17386a6223490d5b8f08d031999b0c0b upstream.
The current code computes a number of channels per SRP target and spreads
them equally across all online NUMA nodes. Each channel is then assigned
a CPU within this node.
In the case of unbalanced, or even unpopulated nodes, some channels do not
get a CPU associated and thus do not get connected. This causes the SRP
connection to fail.
This patch solves the issue by rewriting channel computation and
allocation:
- Drop channel to node/CPU association as it had no real effect on
locality but added unnecessary complexity.
- Tweak the number of channels allocated to reduce CPU contention when
possible:
- Up to one channel per CPU (instead of up to 4 by node)
- At least 4 channels per node, unless ch_count module parameter is
used.
Link: https://lore.kernel.org/r/9cb4d9d3-30ad-2276-7eff-e85f7ddfb411@suse.com
Signed-off-by: Nicolas Morey-Chaisemartin <nmoreychaisemartin@suse.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/ulp/srp/ib_srp.c | 110 ++++++++++++----------------
1 file changed, 45 insertions(+), 65 deletions(-)
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 5492b66a8153..31f8aa2c40ed 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -3628,7 +3628,7 @@ static ssize_t srp_create_target(struct device *dev,
struct srp_rdma_ch *ch;
struct srp_device *srp_dev = host->srp_dev;
struct ib_device *ibdev = srp_dev->dev;
- int ret, node_idx, node, cpu, i;
+ int ret, i, ch_idx;
unsigned int max_sectors_per_mr, mr_per_cmd = 0;
bool multich = false;
uint32_t max_iu_len;
@@ -3753,81 +3753,61 @@ static ssize_t srp_create_target(struct device *dev,
goto out;
ret = -ENOMEM;
- if (target->ch_count == 0)
+ if (target->ch_count == 0) {
target->ch_count =
- max_t(unsigned int, num_online_nodes(),
- min(ch_count ?:
- min(4 * num_online_nodes(),
- ibdev->num_comp_vectors),
- num_online_cpus()));
+ min(ch_count ?:
+ max(4 * num_online_nodes(),
+ ibdev->num_comp_vectors),
+ num_online_cpus());
+ }
+
target->ch = kcalloc(target->ch_count, sizeof(*target->ch),
GFP_KERNEL);
if (!target->ch)
goto out;
- node_idx = 0;
- for_each_online_node(node) {
- const int ch_start = (node_idx * target->ch_count /
- num_online_nodes());
- const int ch_end = ((node_idx + 1) * target->ch_count /
- num_online_nodes());
- const int cv_start = node_idx * ibdev->num_comp_vectors /
- num_online_nodes();
- const int cv_end = (node_idx + 1) * ibdev->num_comp_vectors /
- num_online_nodes();
- int cpu_idx = 0;
-
- for_each_online_cpu(cpu) {
- if (cpu_to_node(cpu) != node)
- continue;
- if (ch_start + cpu_idx >= ch_end)
- continue;
- ch = &target->ch[ch_start + cpu_idx];
- ch->target = target;
- ch->comp_vector = cv_start == cv_end ? cv_start :
- cv_start + cpu_idx % (cv_end - cv_start);
- spin_lock_init(&ch->lock);
- INIT_LIST_HEAD(&ch->free_tx);
- ret = srp_new_cm_id(ch);
- if (ret)
- goto err_disconnect;
+ for (ch_idx = 0; ch_idx < target->ch_count; ++ch_idx) {
+ ch = &target->ch[ch_idx];
+ ch->target = target;
+ ch->comp_vector = ch_idx % ibdev->num_comp_vectors;
+ spin_lock_init(&ch->lock);
+ INIT_LIST_HEAD(&ch->free_tx);
+ ret = srp_new_cm_id(ch);
+ if (ret)
+ goto err_disconnect;
- ret = srp_create_ch_ib(ch);
- if (ret)
- goto err_disconnect;
+ ret = srp_create_ch_ib(ch);
+ if (ret)
+ goto err_disconnect;
- ret = srp_alloc_req_data(ch);
- if (ret)
- goto err_disconnect;
+ ret = srp_alloc_req_data(ch);
+ if (ret)
+ goto err_disconnect;
- ret = srp_connect_ch(ch, max_iu_len, multich);
- if (ret) {
- char dst[64];
-
- if (target->using_rdma_cm)
- snprintf(dst, sizeof(dst), "%pIS",
- &target->rdma_cm.dst);
- else
- snprintf(dst, sizeof(dst), "%pI6",
- target->ib_cm.orig_dgid.raw);
- shost_printk(KERN_ERR, target->scsi_host,
- PFX "Connection %d/%d to %s failed\n",
- ch_start + cpu_idx,
- target->ch_count, dst);
- if (node_idx == 0 && cpu_idx == 0) {
- goto free_ch;
- } else {
- srp_free_ch_ib(target, ch);
- srp_free_req_data(target, ch);
- target->ch_count = ch - target->ch;
- goto connected;
- }
- }
+ ret = srp_connect_ch(ch, max_iu_len, multich);
+ if (ret) {
+ char dst[64];
- multich = true;
- cpu_idx++;
+ if (target->using_rdma_cm)
+ snprintf(dst, sizeof(dst), "%pIS",
+ &target->rdma_cm.dst);
+ else
+ snprintf(dst, sizeof(dst), "%pI6",
+ target->ib_cm.orig_dgid.raw);
+ shost_printk(KERN_ERR, target->scsi_host,
+ PFX "Connection %d/%d to %s failed\n",
+ ch_idx,
+ target->ch_count, dst);
+ if (ch_idx == 0) {
+ goto free_ch;
+ } else {
+ srp_free_ch_ib(target, ch);
+ srp_free_req_data(target, ch);
+ target->ch_count = ch - target->ch;
+ goto connected;
+ }
}
- node_idx++;
+ multich = true;
}
connected:
--
2.21.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] RDMA/srp: Fix support for unpopulated and unbalanced NUMA nodes
2021-03-17 7:45 [PATCH] RDMA/srp: Fix support for unpopulated and unbalanced NUMA nodes Yi Zhang
@ 2021-03-19 9:49 ` Greg KH
0 siblings, 0 replies; 4+ messages in thread
From: Greg KH @ 2021-03-19 9:49 UTC (permalink / raw)
To: Yi Zhang; +Cc: stable, jgg, Nicolas Morey-Chaisemartin, Bart Van Assche
On Wed, Mar 17, 2021 at 03:45:34PM +0800, Yi Zhang wrote:
> From: Nicolas Morey-Chaisemartin <nmoreychaisemartin@suse.com>
>
> This patch fixed one kernel NULL pointer issue with blktests srp/005
Now queued up to 5.11.y and 5.10.y, if you want it in older kernels,
please provide a working backport.
thanks!
greg k-h
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] RDMA/srp: Fix support for unpopulated and unbalanced NUMA nodes
2021-03-15 3:30 ` Yi Zhang
@ 2021-03-15 7:56 ` Greg KH
0 siblings, 0 replies; 4+ messages in thread
From: Greg KH @ 2021-03-15 7:56 UTC (permalink / raw)
To: Yi Zhang
Cc: Jason Gunthorpe, Nicolas Morey-Chaisemartin, linux-rdma,
bart.vanassche, stable
On Mon, Mar 15, 2021 at 11:30:34AM +0800, Yi Zhang wrote:
> Hello
>
> I reproduced the issue on 5.11.7-rc1, could we port this patch to stable
> branch.
<formletter>
This is not the correct way to submit patches for inclusion in the
stable kernel tree. Please read:
https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
for how to do this properly.
</formletter>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] RDMA/srp: Fix support for unpopulated and unbalanced NUMA nodes
[not found] ` <20210217133740.GA2296847@nvidia.com>
@ 2021-03-15 3:30 ` Yi Zhang
2021-03-15 7:56 ` Greg KH
0 siblings, 1 reply; 4+ messages in thread
From: Yi Zhang @ 2021-03-15 3:30 UTC (permalink / raw)
To: Jason Gunthorpe, Nicolas Morey-Chaisemartin
Cc: linux-rdma, bart.vanassche, stable
Hello
I reproduced the issue on 5.11.7-rc1, could we port this patch to stable
branch.
Thanks
Yi
On 2/17/21 9:37 PM, Jason Gunthorpe wrote:
> On Fri, Feb 05, 2021 at 09:14:28AM +0100, Nicolas Morey-Chaisemartin wrote:
>> The current code computes a number of channels per SRP target and spreads
>> them equally across all online NUMA nodes.
>> Each channel is then assigned a CPU within this node.
>>
>> In the case of unbalanced, or even unpopulated nodes, some channels
>> do not get a CPU associated and thus do not get connected.
>> This causes the SRP connection to fail.
>>
>> This patch solves the issue by rewriting channel computation and allocation:
>> - Drop channel to node/CPU association as it had
>> no real effect on locality but added unnecessary complexity.
>> - Tweak the number of channels allocated to reduce CPU contention when possible:
>> - Up to one channel per CPU (instead of up to 4 by node)
>> - At least 4 channels per node, unless ch_count module parameter is used.
>>
>> Signed-off-by: Nicolas Morey-Chaisemartin <nmoreychaisemartin@suse.com>
>> Reviewed-by: Bart Van Assche <bvanassche@acm.org>
>> ---
>> drivers/infiniband/ulp/srp/ib_srp.c | 110 ++++++++++++----------------
>> 1 file changed, 45 insertions(+), 65 deletions(-)
> Applied to for-next, thanks
>
> Jason
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-03-19 9:50 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-17 7:45 [PATCH] RDMA/srp: Fix support for unpopulated and unbalanced NUMA nodes Yi Zhang
2021-03-19 9:49 ` Greg KH
[not found] <9cb4d9d3-30ad-2276-7eff-e85f7ddfb411@suse.com>
[not found] ` <20210217133740.GA2296847@nvidia.com>
2021-03-15 3:30 ` Yi Zhang
2021-03-15 7:56 ` Greg KH
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).