linux-rdma.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Weihang Li <liweihang@huawei.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: <dledford@redhat.com>, <leon@kernel.org>,
	<linux-rdma@vger.kernel.org>, <linuxarm@huawei.com>
Subject: Re: [PATCH for-next] RDMA/hns: Add support for extended atomic in userspace
Date: Thu, 16 Jan 2020 12:05:13 +0800	[thread overview]
Message-ID: <9b7a3629-0564-6643-f6e7-c2f098afd010@huawei.com> (raw)
In-Reply-To: <20200115205611.GG25201@ziepe.ca>



On 2020/1/16 4:56, Jason Gunthorpe wrote:
> On Wed, Jan 15, 2020 at 09:42:26AM +0800, Weihang Li wrote:
>> From: Jiaran Zhang <zhangjiaran@huawei.com>
>>
>> To support extended atomic operations including cmp & swap and fetch & add
>> of 8 bytes, 16 bytes, 32 bytes, 64 bytes in userspace, some field in qpc
>> should be configured.
>>
>> Signed-off-by: Jiaran Zhang <zhangjiaran@huawei.com>
>> Signed-off-by: Weihang Li <liweihang@huawei.com>
>>  drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 16 +++++++++++++++-
>>  drivers/infiniband/hw/hns/hns_roce_hw_v2.h |  3 ++-
>>  2 files changed, 17 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
>> index f1e0ba6..7edf3d8 100644
>> +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
>> @@ -1692,7 +1692,7 @@ static int hns_roce_v2_profile(struct hns_roce_dev *hr_dev)
>>  	caps->max_srq_desc_sz	= HNS_ROCE_V2_MAX_SRQ_DESC_SZ;
>>  	caps->qpc_entry_sz	= HNS_ROCE_V2_QPC_ENTRY_SZ;
>>  	caps->irrl_entry_sz	= HNS_ROCE_V2_IRRL_ENTRY_SZ;
>> -	caps->trrl_entry_sz	= HNS_ROCE_V2_TRRL_ENTRY_SZ;
>> +	caps->trrl_entry_sz	= HNS_ROCE_V2_EXT_ATOMIC_TRRL_ENTRY_SZ;
>>  	caps->cqc_entry_sz	= HNS_ROCE_V2_CQC_ENTRY_SZ;
>>  	caps->srqc_entry_sz	= HNS_ROCE_V2_SRQC_ENTRY_SZ;
>>  	caps->mtpt_entry_sz	= HNS_ROCE_V2_MTPT_ENTRY_SZ;
>> @@ -3286,6 +3286,9 @@ static void set_access_flags(struct hns_roce_qp *hr_qp,
>>  	roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
>>  		     !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
>>  	roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S, 0);
>> +	roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_EXT_ATE_S,
>> +		     !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
>> +	roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_EXT_ATE_S, 0);
>>  }
>>  
>>  static void set_qpc_wqe_cnt(struct hns_roce_qp *hr_qp,
>> @@ -3653,6 +3656,12 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp,
>>  			     IB_ACCESS_REMOTE_ATOMIC));
>>  		roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
>>  			     0);
>> +		roce_set_bit(context->byte_76_srqn_op_en,
>> +			     V2_QPC_BYTE_76_EXT_ATE_S,
>> +			     !!(attr->qp_access_flags &
>> +				IB_ACCESS_REMOTE_ATOMIC));
>> +		roce_set_bit(qpc_mask->byte_76_srqn_op_en,
>> +			     V2_QPC_BYTE_76_EXT_ATE_S, 0);
>>  	} else {
>>  		roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RRE_S,
>>  			     !!(hr_qp->access_flags & IB_ACCESS_REMOTE_READ));
>> @@ -3668,6 +3677,11 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp,
>>  			     !!(hr_qp->access_flags & IB_ACCESS_REMOTE_ATOMIC));
>>  		roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
>>  			     0);
>> +		roce_set_bit(context->byte_76_srqn_op_en,
>> +			     V2_QPC_BYTE_76_EXT_ATE_S,
>> +			     !!(hr_qp->access_flags & IB_ACCESS_REMOTE_ATOMIC));
>> +		roce_set_bit(qpc_mask->byte_76_srqn_op_en,
>> +			     V2_QPC_BYTE_76_EXT_ATE_S, 0);
>>  	}
> 
> What happens to your userspace if it runs on an old kernel and tries
> to use extended atomic?
> 
> Jason
>

Hi Jason,

If the hns userspace runs with old kernel, the hardware will report a asynchronous
event for the extended atomic operation and modify the qp to error state because
the enable bit in this qp's context hasn't been set.

The driver will print like this:

[ 1252.240921] hns3 0000:7d:00.0: Invalid request local work queue 0x9 error.
[ 1252.247772] hns3 0000:7d:00.0: no hr_qp can be found!

Thanks
Weihang

> .
> 


  reply	other threads:[~2020-01-16  4:05 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-15  1:42 [PATCH for-next] RDMA/hns: Add support for extended atomic in userspace Weihang Li
2020-01-15 20:56 ` Jason Gunthorpe
2020-01-16  4:05   ` Weihang Li [this message]
2020-01-16 19:51     ` Jason Gunthorpe
2020-01-22  8:54       ` Weihang Li
2020-01-22 14:08         ` Tom Talpey
2020-01-26  3:38           ` Weihang Li
2020-01-23 22:54         ` Jason Gunthorpe
2020-01-26  3:42           ` Weihang Li

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9b7a3629-0564-6643-f6e7-c2f098afd010@huawei.com \
    --to=liweihang@huawei.com \
    --cc=dledford@redhat.com \
    --cc=jgg@ziepe.ca \
    --cc=leon@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=linuxarm@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).