Re: [PATCH] nvme-pci: slimmer CQ head update

From: Will Deacon <will@kernel.org>
To: Keith Busch <kbusch@kernel.org>
Cc: sagi@grimberg.me, John Garry <john.garry@huawei.com>,
	linux-nvme@lists.infradead.org, Christoph Hellwig <hch@lst.de>,
	axboe@fb.com, Robin Murphy <robin.murphy@arm.com>,
	Alexey Dobriyan <adobriyan@gmail.com>
Subject: Re: [PATCH] nvme-pci: slimmer CQ head update
Date: Fri, 8 May 2020 12:40:20 +0100	[thread overview]
Message-ID: <20200508114020.GA16247@willie-the-truck> (raw)
In-Reply-To: <20200507180612.GD2621480@dhcp-10-100-145-180.wdl.wdc.com>

On Thu, May 07, 2020 at 11:06:12AM -0700, Keith Busch wrote:
> On Thu, May 07, 2020 at 06:44:29PM +0100, Will Deacon wrote:
> > On Thu, May 07, 2020 at 10:35:45AM -0700, Keith Busch wrote:
> > > 
> > > Yeah, that might be something to consider. If that's the case, then it
> > > should be reproducible with a different vendor's nvme controller.
> > > 
> > > If the behavior is unique to this particular nvme model, then it would
> > > sound like the controller is creating a CQE in multiple memory writes
> > > either high-to-low, or with Relaxed Ordering enabled in the TLPs.
> > 
> > [disclaimer: I don't know anything about nvme, so this might be way off! I
> > know a bit about the Arm memory model though, so I can help with that side.]
> > 
> > Are you sure there's an ordering requirement? My reading of the quote
> > from the spec is that "last write" just means the final write, not
> > necessarily the word at the highest address.
> 
> Oh, NVMe's phase bit is located in the last word of a completion entry,
> so the final write of a completion entry must be the highest address of
> that entry.

Thanks. It sounds like it would still be valid (but weird?) for the device
to do: <write CQE[n-2] ... CQE[0]>; <write CQE[n-1]> though.

> > > But if some other interaction with the arm memory model is suspect, I'm
> > > not sure how to confirm or debug.
> > 
> > From what you described, it sounds like the flow should go something like:
> > 
> >   <Read the word of the CQE containing the updated Phase Tag Bit>
> >   dma_rmb();
> >   <Read the other words of the CQE>
> > 
> > How easy is it to check if that sort of thing is being followed?
> 
> We don't have such a barrier in the driver. The phase must be written
> last, so having it visible to the CPU was supposed to guarantee previous
> DMA writes are also visible. Is this not the case?

Without a barrier on the read side, something like the following might be
able to happen:

	CPU				NVME
	Read stale data
					Write data
					Write phase bit
	Read phase bit

So you need a dma_rmb() on the CPU side to prevent that reordering, and
ensure that the phase bit is read first. Anyway, the only reason I mentioned
it is because it should  be relatively straightforward to add the barrier
(famous last words...) and see if it helps with the problem. If not, we're
looking in the wrong place.

> FWIW, most NVMe controllers create the entire 16-byte CQE with a single
> DMA write.

Yes, that definitely sounds like the sane approach. Perhaps John will be
able to confirm what his device does.

Will

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme