Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Jason Gunthorpe <jgg@ziepe.ca>
To: Weihang Li <liweihang@huawei.com>
Cc: dledford@redhat.com, leon@kernel.org, linux-rdma@vger.kernel.org,
	linuxarm@huawei.com
Subject: Re: [PATCH for-next] RDMA/hns: Add support for extended atomic in userspace
Date: Thu, 16 Jan 2020 15:51:18 -0400
Message-ID: <20200116195118.GG10759@ziepe.ca> (raw)
In-Reply-To: <9b7a3629-0564-6643-f6e7-c2f098afd010@huawei.com>

On Thu, Jan 16, 2020 at 12:05:13PM +0800, Weihang Li wrote:
> 
> 
> On 2020/1/16 4:56, Jason Gunthorpe wrote:
> > On Wed, Jan 15, 2020 at 09:42:26AM +0800, Weihang Li wrote:
> >> From: Jiaran Zhang <zhangjiaran@huawei.com>
> >>
> >> To support extended atomic operations including cmp & swap and fetch & add
> >> of 8 bytes, 16 bytes, 32 bytes, 64 bytes in userspace, some field in qpc
> >> should be configured.
> >>
> >> Signed-off-by: Jiaran Zhang <zhangjiaran@huawei.com>
> >> Signed-off-by: Weihang Li <liweihang@huawei.com>
> >>  drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 16 +++++++++++++++-
> >>  drivers/infiniband/hw/hns/hns_roce_hw_v2.h |  3 ++-
> >>  2 files changed, 17 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >> index f1e0ba6..7edf3d8 100644
> >> +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >> @@ -1692,7 +1692,7 @@ static int hns_roce_v2_profile(struct hns_roce_dev *hr_dev)
> >>  	caps->max_srq_desc_sz	= HNS_ROCE_V2_MAX_SRQ_DESC_SZ;
> >>  	caps->qpc_entry_sz	= HNS_ROCE_V2_QPC_ENTRY_SZ;
> >>  	caps->irrl_entry_sz	= HNS_ROCE_V2_IRRL_ENTRY_SZ;
> >> -	caps->trrl_entry_sz	= HNS_ROCE_V2_TRRL_ENTRY_SZ;
> >> +	caps->trrl_entry_sz	= HNS_ROCE_V2_EXT_ATOMIC_TRRL_ENTRY_SZ;
> >>  	caps->cqc_entry_sz	= HNS_ROCE_V2_CQC_ENTRY_SZ;
> >>  	caps->srqc_entry_sz	= HNS_ROCE_V2_SRQC_ENTRY_SZ;
> >>  	caps->mtpt_entry_sz	= HNS_ROCE_V2_MTPT_ENTRY_SZ;
> >> @@ -3286,6 +3286,9 @@ static void set_access_flags(struct hns_roce_qp *hr_qp,
> >>  	roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
> >>  		     !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
> >>  	roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S, 0);
> >> +	roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_EXT_ATE_S,
> >> +		     !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
> >> +	roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_EXT_ATE_S, 0);
> >>  }
> >>  
> >>  static void set_qpc_wqe_cnt(struct hns_roce_qp *hr_qp,
> >> @@ -3653,6 +3656,12 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp,
> >>  			     IB_ACCESS_REMOTE_ATOMIC));
> >>  		roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
> >>  			     0);
> >> +		roce_set_bit(context->byte_76_srqn_op_en,
> >> +			     V2_QPC_BYTE_76_EXT_ATE_S,
> >> +			     !!(attr->qp_access_flags &
> >> +				IB_ACCESS_REMOTE_ATOMIC));
> >> +		roce_set_bit(qpc_mask->byte_76_srqn_op_en,
> >> +			     V2_QPC_BYTE_76_EXT_ATE_S, 0);
> >>  	} else {
> >>  		roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RRE_S,
> >>  			     !!(hr_qp->access_flags & IB_ACCESS_REMOTE_READ));
> >> @@ -3668,6 +3677,11 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp,
> >>  			     !!(hr_qp->access_flags & IB_ACCESS_REMOTE_ATOMIC));
> >>  		roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
> >>  			     0);
> >> +		roce_set_bit(context->byte_76_srqn_op_en,
> >> +			     V2_QPC_BYTE_76_EXT_ATE_S,
> >> +			     !!(hr_qp->access_flags & IB_ACCESS_REMOTE_ATOMIC));
> >> +		roce_set_bit(qpc_mask->byte_76_srqn_op_en,
> >> +			     V2_QPC_BYTE_76_EXT_ATE_S, 0);
> >>  	}
> > 
> > What happens to your userspace if it runs on an old kernel and tries
> > to use extended atomic?
> > 
> > Jason
> >
> 
> Hi Jason,
> 
> If the hns userspace runs with old kernel, the hardware will report a asynchronous
> event for the extended atomic operation and modify the qp to error state because
> the enable bit in this qp's context hasn't been set.
> 
> The driver will print like this:
> 
> [ 1252.240921] hns3 0000:7d:00.0: Invalid request local work queue 0x9 error.
> [ 1252.247772] hns3 0000:7d:00.0: no hr_qp can be found!

Ideally the provider will not set
IBV_PCI_ATOMIC_OPERATION_4_BYTE_SIZE_SUP and related without kernel
support..

I've applied this patch, but I feel like you may need a followup to
fix the capability reporting?

Jason

      reply index

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-15  1:42 Weihang Li
2020-01-15 20:56 ` Jason Gunthorpe
2020-01-16  4:05   ` Weihang Li
2020-01-16 19:51     ` Jason Gunthorpe [this message]

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200116195118.GG10759@ziepe.ca \
    --to=jgg@ziepe.ca \
    --cc=dledford@redhat.com \
    --cc=leon@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=linuxarm@huawei.com \
    --cc=liweihang@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git