Re: [PATCH] nvme-pci: slimmer CQ head update

[Keith Busch <kbusch@kernel.org>]

On Thu, May 07, 2020 at 06:44:29PM +0100, Will Deacon wrote:
> On Thu, May 07, 2020 at 10:35:45AM -0700, Keith Busch wrote:
> > 
> > Yeah, that might be something to consider. If that's the case, then it
> > should be reproducible with a different vendor's nvme controller.
> > 
> > If the behavior is unique to this particular nvme model, then it would
> > sound like the controller is creating a CQE in multiple memory writes
> > either high-to-low, or with Relaxed Ordering enabled in the TLPs.
> 
> [disclaimer: I don't know anything about nvme, so this might be way off! I
> know a bit about the Arm memory model though, so I can help with that side.]
> 
> Are you sure there's an ordering requirement? My reading of the quote
> from the spec is that "last write" just means the final write, not
> necessarily the word at the highest address.

Oh, NVMe's phase bit is located in the last word of a completion entry,
so the final write of a completion entry must be the highest address of
that entry.
 
> > But if some other interaction with the arm memory model is suspect, I'm
> > not sure how to confirm or debug.
> 
> From what you described, it sounds like the flow should go something like:
> 
>   <Read the word of the CQE containing the updated Phase Tag Bit>
>   dma_rmb();
>   <Read the other words of the CQE>
> 
> How easy is it to check if that sort of thing is being followed?

We don't have such a barrier in the driver. The phase must be written
last, so having it visible to the CPU was supposed to guarantee previous
DMA writes are also visible. Is this not the case?

FWIW, most NVMe controllers create the entire 16-byte CQE with a single
DMA write.

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme