Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Honggang LI <honli@redhat.com>
To: Mike Marciniszyn <mike.marciniszyn@intel.com>
Cc: jgg@ziepe.ca, dledford@redhat.com, linux-rdma@vger.kernel.org
Subject: Re: [PATCH for-rc] IB/rdmavt: Fix RQ counting issues causing use of an invalid RWQE
Date: Wed, 29 Jul 2020 22:33:26 +0800
Message-ID: <20200729143326.GA2493772@dhcp-128-72.nay.redhat.com> (raw)
In-Reply-To: <20200728183848.22226.29132.stgit@awfm-01.aw.intel.com>

On Tue, Jul 28, 2020 at 02:38:48PM -0400, Mike Marciniszyn wrote:
> The lookaside count is improperly initialized to the size of the
> Receive Queue with the additional +1.  In the traces below, the
> RQ size is 384, so the count was set to 385.
> 
> The lookaside count is then rarely refreshed.  Note the high and
> incorrect count in the trace below:
> 
> rvt_get_rwqe: [hfi1_0] wqe ffffc900078e9008 wr_id 55c7206d75a0 qpn c
> 	qpt 2 pid 3018 num_sge 1 head 1 tail 0, count 385
> rvt_get_rwqe: (hfi1_rc_rcv+0x4eb/0x1480 [hfi1] <- rvt_get_rwqe) ret=0x1
> 
> The head,tail indicate there is only one RWQE posted although the count
> says 385 and we correctly return the element 0.
> 
> The next call to rvt_get_rwqe with the decremented count:
> 
> rvt_get_rwqe: [hfi1_0] wqe ffffc900078e9058 wr_id 0 qpn c
> 	qpt 2 pid 3018 num_sge 0 head 1 tail 1, count 384
> rvt_get_rwqe: (hfi1_rc_rcv+0x4eb/0x1480 [hfi1] <- rvt_get_rwqe) ret=0x1
> 
> Note that the RQ is empty (head == tail) yet we return the RWQE at tail 1,
> which is not valid because of the bogus high count.
> 
> Best case, the RWQE has never been posted and the rc logic sees an RWQE
> that is too small (all zeros) and puts the QP into an error state.
> 
> In the worst case, a server slow at posting receive buffers might fool
> rvt_get_rwqe() into fetching an old RWQE and corrupt memory.
> 
> Fix by deleting the faulty initialization code and creating an
> inline to fetch the posted count and convert all callers to use
> new inline.
> 
> Fixes: f592ae3c999f ("IB/rdmavt: Fracture single lock used for posting and processing RWQEs")

Confirmed this patch works for me. Thanks

Tested-by: Honggang Li <honli@redhat.com>


  reply index

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-28 18:38 Mike Marciniszyn
2020-07-29 14:33 ` Honggang LI [this message]
2020-07-29 18:54 ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200729143326.GA2493772@dhcp-128-72.nay.redhat.com \
    --to=honli@redhat.com \
    --cc=dledford@redhat.com \
    --cc=jgg@ziepe.ca \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mike.marciniszyn@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git