Re: ioremap_uc() followed by set_memory_wc() - burrying MTRR

[Hyong-Youb Kim <hkim@cspi.com>]

On Thu, Apr 16, 2015 at 01:58:16AM +0200, Luis R. Rodriguez wrote:
> 
> An alternative... is to just ioremap_wc() the entire region, including
> MMIO registers for these old devices. I see one ethernet driver that does
> this, myri10ge, and am curious how and why they ended up deciding this
> and if they have run into any issues. I wonder if this is a reasonable
> comrpomise for these 2 remaining corner cases.
> 

For myri10ge, it a performance thing.  Descriptor rings are in NIC
memory BAR0, not in host memory.  Say, to send a packet, the driver
writes the send descriptor to the ioremap'd NIC memory.  It is a
multi-word descriptor.  So, to send it as one PCIE MWr transaction,
the driver maps the whole BAR0 as WC and does "copy descriptor; wmb".

Without WC, descriptors would end up as multiple 4B or 8B MWr packets
to the NIC, which has a pretty big performance impact on this
particular NIC.

Most registers that do not want WC are actually in BAR2, which is not
mapped as WC.  For registers that are in BAR0, we do "write to the
register; wmb".  If we want to wait till the NIC has seen the write,
we do "write; wmb; read".

This approach has worked for this device for many years.  I cannot say
whether it works for other devices, though.