Re: ioremap_uc() followed by set_memory_wc() - burrying MTRR

["Luis R. Rodriguez" <mcgrof@suse.com>]

On Tue, Apr 21, 2015 at 04:57:32PM -0600, Jason Gunthorpe wrote:
> On Wed, Apr 22, 2015 at 12:46:01AM +0200, Luis R. Rodriguez wrote:
> 
> > are talking about annotating the qib driver as "known to be broken without PAT"
> > and since the ipath driver needs considerable work to be ported to
> > use PAT (the
> 
> This only seems to be true for one of the chips that driver supports,
> not all possibilities.
> 
> > userspace register is just one area) I wanted to review if we can just remove
> > MTRR use on the ipath driver and annotate write-combining with PAT as a TODO
> > item.
> 
> AFAIK, dropping MTRR support will completely break the performance to
> the point the driver is unusable. If we drop MTRR we may as well
> remove the driver.

To be clear, the arch_phys_wc_add() API will be a no-op when PAT is
enabled on a system. Although some folks think PAT is new, its not,
its just that our implementation on Linux lacked a bit of push, recent
changes however make PAT support complete and that means now we'll have
PAT enabled on systems that likely didn't before on recent kernels.

There are other important motivations to use PAT:

 * Xen won't work with MTRR, having a unified PAT enabled kernel
   will ensure that when Xen is used write-combinging will take
   effect

 * Long term we want to make strong UC the default to ioremap() on
   x86, right now its not, its UC-, we need to convert all drivers
   that want write-combining over to use ioremap_wc() for their
   specific area, and it must not overlap. There are issues with
   using mtrr_add() on regions marked as UC-, since its the default
   this  means that mtrr_add() use on ioramp'd areas on PAT systems
   will actually make write-combining *void*. Refer to this table
   for combinatorial effects for non-PAT / PAT of wc MTRR:

   https://marc.info/?l=linux-kernel&m=142964809710517&w=1

So as the train of PAT enablement moves forward it means systems
that have PAT enabled now might not have write-combining effective.
In order to get the best of both worlds, non-PAT and PAT systems
we must design drivers cleanly for the non-writecombining and
write-combining areas. This mean using ioremap_nocache() for MMIO
and ioremap_wc() *only* for the desired, write-combining area,
followed by arch_phys_wc_add().

> Mike, do you think the time is right to just remove the iPath driver?

With PAT now being default the driver effectively won't work
with write-combining on modern kernels. Even if systems are old
they likely had PAT support, when upgrading kernels PAT will work
but write-combing won't on ipath.

  Luis