[PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

This patch adds the Write-through memory type in combination with mtrr.
If you call ioremap_cache to request cachable memory (write-back) the
function tries to set the PAT to write-back only if the mtrr setting of
the requested region is also marked as Write-Back.
If the mttr regions are marked e.g. as Write-through or with other
types, the function will always return UC- memory.

If you check the Intel document " IA-32 SDM vol 3a table Effective
Memory Type", there
are many other combinations possible.

This patch will only add the following combination:
  PAT=Write-Back + MTRR=Write-Through = Effective Memory of
Write-Through

Tested on - Intel (R) Atom E680 (Tunnel Creek)
          - Intel (R) Core(TM)2 Duo

Signed-off-by: Andreas Werner <wernerandy@gmx.de>
---
 arch/x86/mm/pat.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/mm/pat.c b/arch/x86/mm/pat.c
index 6574388..9cfe107 100644
--- a/arch/x86/mm/pat.c
+++ b/arch/x86/mm/pat.c
@@ -149,10 +149,15 @@ static unsigned long pat_x_mtrr_type(u64 start, u64 end, unsigned long req_type)
 		u8 mtrr_type;
 
 		mtrr_type = mtrr_type_lookup(start, end);
-		if (mtrr_type != MTRR_TYPE_WRBACK)
-			return _PAGE_CACHE_UC_MINUS;
 
-		return _PAGE_CACHE_WB;
+		switch (mtrr_type) {
+		case MTRR_TYPE_WRBACK:
+		case MTRR_TYPE_WRTHROUGH:
+		  return _PAGE_CACHE_WB;
+
+		default:
+		  return _PAGE_CACHE_UC_MINUS;
+		}
 	}
 
 	return req_type;
-- 
1.8.4

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Borislav Petkov]

On Sun, Oct 27, 2013 at 01:55:25PM +0100, Andreas Werner wrote:
> This patch adds the Write-through memory type in combination with mtrr.
> If you call ioremap_cache to request cachable memory (write-back) the
> function tries to set the PAT to write-back only if the mtrr setting of
> the requested region is also marked as Write-Back.
> If the mttr regions are marked e.g. as Write-through or with other
> types, the function will always return UC- memory.
> 
> If you check the Intel document " IA-32 SDM vol 3a table Effective
> Memory Type", there
> are many other combinations possible.
> 
> This patch will only add the following combination:
>   PAT=Write-Back + MTRR=Write-Through = Effective Memory of
> Write-Through

And yet the code below returns WB for WT MTRR type which can't be
right since having a write-through mapping cannot be compatible with
write-back as the last caches writes into the cache instead of writing
them through to memory.

Why do you even care about WT? Are you trying to fix a bug or what?

> Tested on - Intel (R) Atom E680 (Tunnel Creek)
>           - Intel (R) Core(TM)2 Duo
> 
> Signed-off-by: Andreas Werner <wernerandy@gmx.de>
> ---
>  arch/x86/mm/pat.c | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/mm/pat.c b/arch/x86/mm/pat.c
> index 6574388..9cfe107 100644
> --- a/arch/x86/mm/pat.c
> +++ b/arch/x86/mm/pat.c
> @@ -149,10 +149,15 @@ static unsigned long pat_x_mtrr_type(u64 start, u64 end, unsigned long req_type)
>  		u8 mtrr_type;
>  
>  		mtrr_type = mtrr_type_lookup(start, end);
> -		if (mtrr_type != MTRR_TYPE_WRBACK)
> -			return _PAGE_CACHE_UC_MINUS;
>  
> -		return _PAGE_CACHE_WB;
> +		switch (mtrr_type) {
> +		case MTRR_TYPE_WRBACK:
> +		case MTRR_TYPE_WRTHROUGH:
> +		  return _PAGE_CACHE_WB;
> +
> +		default:
> +		  return _PAGE_CACHE_UC_MINUS;
> +		}
>  	}
>  
>  	return req_type;

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Sun, Oct 27, 2013 at 02:34:01PM +0100, Borislav Petkov wrote:
> On Sun, Oct 27, 2013 at 01:55:25PM +0100, Andreas Werner wrote:
> > This patch adds the Write-through memory type in combination with mtrr.
> > If you call ioremap_cache to request cachable memory (write-back) the
> > function tries to set the PAT to write-back only if the mtrr setting of
> > the requested region is also marked as Write-Back.
> > If the mttr regions are marked e.g. as Write-through or with other
> > types, the function will always return UC- memory.
> > 
> > If you check the Intel document " IA-32 SDM vol 3a table Effective
> > Memory Type", there
> > are many other combinations possible.
> > 
> > This patch will only add the following combination:
> >   PAT=Write-Back + MTRR=Write-Through = Effective Memory of
> > Write-Through
> 
> And yet the code below returns WB for WT MTRR type which can't be
> right since having a write-through mapping cannot be compatible with
> write-back as the last caches writes into the cache instead of writing
> them through to memory.
> 
> Why do you even care about WT? Are you trying to fix a bug or what?
>

Im currently working on an ethernet driver for our own ETH core.
The problem is that one requirement is to not use DMA to transmit 
or receive the data. This means the that the ethernet buffers are not 
located in the main memory. They are located in the FPGA internal RAM.
 
To transmit or receive a frame, i have to read or write to mmio to get the data.
 
Intel has introduced the instruction "clflush" which can flush a cache line.
I want to use the caches for those mmio (eth buffer) to speed up the transmit/receive
and to transmit/receive using PCIe bursts (read/write).
 
The problem was if i set the buffer to Write-Back and call clflush on 
those mmio-addresses, the system crashed without any output.
I found this articel 
http://software.intel.com/en-us/forums/topic/393070
 
After that i configured the transmit buffers to be Write-Combining (only write to that adresses) 
using ioremap_wc, and the receive buffers to be Write-Through (ioremap_cache + mtrr Write-Through + this kernel patch) 
everything worked as expected.
 
On PCIe Tracer i can see the bursts on read/write on my device with this configuration.

I know that this is a special use case but there is also one person more out there,
who use that configuration. But why not allow this setting in the kernel? It is
also mentioned in the Intel IA32 document.

I know that there is also an instruction called "movntdqa" but this is not support on my platform
(Intel E680) this instruction was introduced in SSE3. So i can not use it.

If you need more detailed informations please ask me.

Best Regards
Andy

> > Tested on - Intel (R) Atom E680 (Tunnel Creek)
> >           - Intel (R) Core(TM)2 Duo
> > 
> > Signed-off-by: Andreas Werner <wernerandy@gmx.de>
> > ---
> >  arch/x86/mm/pat.c | 11 ++++++++---
> >  1 file changed, 8 insertions(+), 3 deletions(-)
> > 
> > diff --git a/arch/x86/mm/pat.c b/arch/x86/mm/pat.c
> > index 6574388..9cfe107 100644
> > --- a/arch/x86/mm/pat.c
> > +++ b/arch/x86/mm/pat.c
> > @@ -149,10 +149,15 @@ static unsigned long pat_x_mtrr_type(u64 start, u64 end, unsigned long req_type)
> >  		u8 mtrr_type;
> >  
> >  		mtrr_type = mtrr_type_lookup(start, end);
> > -		if (mtrr_type != MTRR_TYPE_WRBACK)
> > -			return _PAGE_CACHE_UC_MINUS;
> >  
> > -		return _PAGE_CACHE_WB;
> > +		switch (mtrr_type) {
> > +		case MTRR_TYPE_WRBACK:
> > +		case MTRR_TYPE_WRTHROUGH:
> > +		  return _PAGE_CACHE_WB;
> > +
> > +		default:
> > +		  return _PAGE_CACHE_UC_MINUS;
> > +		}
> >  	}
> >  
> >  	return req_type;
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> Sent from a fat crate under my desk. Formatting is fine.
> --

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Borislav Petkov]

On Sun, Oct 27, 2013 at 05:51:59PM +0100, Andreas Werner wrote:
> Im currently working on an ethernet driver for our own ETH core. The
> problem is that one requirement is to not use DMA to transmit or
> receive the data. This means the that the ethernet buffers are not
> located in the main memory. They are located in the FPGA internal RAM.
>
> To transmit or receive a frame, i have to read or write to mmio to get
> the data.
>
> Intel has introduced the instruction "clflush" which can flush a cache
> line. I want to use the caches for those mmio (eth buffer) to speed
> up the transmit/receive and to transmit/receive using PCIe bursts
> (read/write).
>
> The problem was if i set the buffer to Write-Back and call clflush on
> those mmio-addresses, the system crashed without any output.

But allocating a WB region and calling CLFLUSH right after writing
into it sounds like you want to allocate an UC region, no? Writing
into it will make sure the data has reached memory and is not in the
cache, basically what CLFLUSH does but by having it UC, this happens
automatically.

So basically what ioremap_nocache does.

> I found this articel 
> http://software.intel.com/en-us/forums/topic/393070
>
> After that i configured the transmit buffers to be Write-Combining
> (only write to that adresses) using ioremap_wc, and the receive
> buffers to be Write-Through (ioremap_cache + mtrr Write-Through + this
> kernel patch) everything worked as expected.

Right, but this all sounds like you want to use ioremap_nocache which
makes your buffers UC.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Sun, Oct 27, 2013 at 06:31:31PM +0100, Borislav Petkov wrote:
> On Sun, Oct 27, 2013 at 05:51:59PM +0100, Andreas Werner wrote:
> > Im currently working on an ethernet driver for our own ETH core. The
> > problem is that one requirement is to not use DMA to transmit or
> > receive the data. This means the that the ethernet buffers are not
> > located in the main memory. They are located in the FPGA internal RAM.
> >
> > To transmit or receive a frame, i have to read or write to mmio to get
> > the data.
> >
> > Intel has introduced the instruction "clflush" which can flush a cache
> > line. I want to use the caches for those mmio (eth buffer) to speed
> > up the transmit/receive and to transmit/receive using PCIe bursts
> > (read/write).
> >
> > The problem was if i set the buffer to Write-Back and call clflush on
> > those mmio-addresses, the system crashed without any output.
> 
> But allocating a WB region and calling CLFLUSH right after writing
> into it sounds like you want to allocate an UC region, no? Writing
> into it will make sure the data has reached memory and is not in the
> cache, basically what CLFLUSH does but by having it UC, this happens
> automatically.
> 
> So basically what ioremap_nocache does.
> 
> > I found this articel 
> > http://software.intel.com/en-us/forums/topic/393070
> >
> > After that i configured the transmit buffers to be Write-Combining
> > (only write to that adresses) using ioremap_wc, and the receive
> > buffers to be Write-Through (ioremap_cache + mtrr Write-Through + this
> > kernel patch) everything worked as expected.
> 
> Right, but this all sounds like you want to use ioremap_nocache which
> makes your buffers UC.
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> Sent from a fat crate under my desk. Formatting is fine.
> --

Maybe you missunderstood me.

My configuration is:

Transmit Buffers WC (only write to that buffer)
i have PICe bursts on my tracer.

Receive Buffers WT (only read to that buffer).
I use clflush_cache_range before reading from that
adresses and i have PCIe bursts on my tracer.

With UC memory there are no PCIe bursts and my bandwidth
is very slow.

Best regards
	Andy

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Borislav Petkov]

On Sun, Oct 27, 2013 at 06:56:08PM +0100, Andreas Werner wrote:
> Transmit Buffers WC (only write to that buffer)
> i have PICe bursts on my tracer.

For that you can do ioremap_wc().

> Receive Buffers WT (only read to that buffer). I use
> clflush_cache_range before reading from that adresses and i have PCIe
> bursts on my tracer.

That one I don't understand - why would you need a WT buffer? It only
caches reads but you will read from it only once after it has been
received. Why pollute the cache?

IOW, you probably could use a WC buffer here too, as it would combine
the writes coming from the FPGA.

Btw, there's also mtrr_add(..., MTRR_TYPE_WRTHROUGH, ) if you must use a
WT thing. Have you tried that?

> With UC memory there are no PCIe bursts and my bandwidth is very slow.

Right.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Sun, Oct 27, 2013 at 08:01:48PM +0100, Borislav Petkov wrote:
> On Sun, Oct 27, 2013 at 06:56:08PM +0100, Andreas Werner wrote:
> > Transmit Buffers WC (only write to that buffer)
> > i have PICe bursts on my tracer.
> 
> For that you can do ioremap_wc().

Yes i am currently using ioremap_wc() and it is working
as expected.
> 
> > Receive Buffers WT (only read to that buffer). I use
> > clflush_cache_range before reading from that adresses and i have PCIe
> > bursts on my tracer.
> 
> That one I don't understand - why would you need a WT buffer? It only
> caches reads but you will read from it only once after it has been
> received. Why pollute the cache?
> 
> IOW, you probably could use a WC buffer here too, as it would combine
> the writes coming from the FPGA.
> 
> Btw, there's also mtrr_add(..., MTRR_TYPE_WRTHROUGH, ) if you must use a
> WT thing. Have you tried that?
> 
For reading i need to map the mmio with attributes that allow cache-line read.
Therefore i use WT. For the Virtual address i use ioremap_cache in combination
with this patch to get an effective memory type of "Write-Through". This allows
me to read from the mmio with "PCIe burst". The write behaviour to this
region do not matter.

The clflush is used to remove stale cache lines from the cache so that
the read operation to a line goes to the MMIO device.
WT was the only one where i had bursts in reading.
A WC buffer had the same behaviour like UC on the PCIe Tracer (for reading).

I use mtrr_add to make an entry in the MTRR with a typ of WRTHROUGH
for the "receive" memory region.


> > With UC memory there are no PCIe bursts and my bandwidth is very slow.
> 
> Right.
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> Sent from a fat crate under my desk. Formatting is fine.
> --

Regards
Andy

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Ingo Molnar]

* Andreas Werner <wernerandy@gmx.de> wrote:

> > IOW, you probably could use a WC buffer here too, as it would 
> > combine the writes coming from the FPGA.
> > 
> > Btw, there's also mtrr_add(..., MTRR_TYPE_WRTHROUGH, ) if you 
> > must use a WT thing. Have you tried that?
> 
> For reading i need to map the mmio with attributes that allow 
> cache-line read. Therefore i use WT. For the Virtual address i use 
> ioremap_cache in combination with this patch to get an effective 
> memory type of "Write-Through". This allows me to read from the 
> mmio with "PCIe burst". The write behaviour to this region do not 
> matter.

And regular write-back cacheable isn't sufficient because the CPU 
could do things like prefetch your range automatically?

If the reads are for packet data and not for commands, WB could 
still be beneficial as it should allow even higher bandwidth. (For 
non-data with real semantics WB is probably not good.)

Thanks,

	Ingo

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Borislav Petkov]

On Mon, Oct 28, 2013 at 11:17:49AM +0100, Ingo Molnar wrote:
> And regular write-back cacheable isn't sufficient because the CPU
> could do things like prefetch your range automatically?

Yeah, he's doing a CLFLUSH anyway which basically makes it a
write-through...

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Ingo Molnar]

* Borislav Petkov <bp@alien8.de> wrote:

> On Mon, Oct 28, 2013 at 11:17:49AM +0100, Ingo Molnar wrote:
>
> > And regular write-back cacheable isn't sufficient because the 
> > CPU could do things like prefetch your range automatically?
> 
> Yeah, he's doing a CLFLUSH anyway which basically makes it a 
> write-through...

The CLFLUSH is done afterwards (making it a use-once thing), so WB 
might still be faster and would avoid the PAT headache ...

Thanks,

	Ingo

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[H. Peter Anvin]

Sort of; the writes combine in the cache.

Borislav Petkov <bp@alien8.de> wrote:
>On Mon, Oct 28, 2013 at 11:17:49AM +0100, Ingo Molnar wrote:
>> And regular write-back cacheable isn't sufficient because the CPU
>> could do things like prefetch your range automatically?
>
>Yeah, he's doing a CLFLUSH anyway which basically makes it a
>write-through...

-- 
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Mon, Oct 28, 2013 at 11:17:49AM +0100, Ingo Molnar wrote:
> 
> * Andreas Werner <wernerandy@gmx.de> wrote:
> 
> > > IOW, you probably could use a WC buffer here too, as it would 
> > > combine the writes coming from the FPGA.
> > > 
> > > Btw, there's also mtrr_add(..., MTRR_TYPE_WRTHROUGH, ) if you 
> > > must use a WT thing. Have you tried that?
> > 
> > For reading i need to map the mmio with attributes that allow 
> > cache-line read. Therefore i use WT. For the Virtual address i use 
> > ioremap_cache in combination with this patch to get an effective 
> > memory type of "Write-Through". This allows me to read from the 
> > mmio with "PCIe burst". The write behaviour to this region do not 
> > matter.
> 
> And regular write-back cacheable isn't sufficient because the CPU 
> could do things like prefetch your range automatically?
> 
> If the reads are for packet data and not for commands, WB could 
> still be beneficial as it should allow even higher bandwidth. (For 
> non-data with real semantics WB is probably not good.)
> 
> Thanks,
> 
> 	Ingo

Yes the reads are only for packet data, the commands or configuration
registers are mapped non cachable.

I´ve tried WB, but on PCIe Tracer i could not see any burst access.
Thats the reason why i have created this patch.

Is there a chance to get this patch into the kernel? Or
is this solution so special?

Regards
Andy

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Borislav Petkov]

On Mon, Oct 28, 2013 at 11:31:32AM +0100, Ingo Molnar wrote:
> The CLFLUSH is done afterwards (making it a use-once thing), so WB
> might still be faster and would avoid the PAT headache ...

Absolutely, that's why there's no need for a WT buffer at all, methinks.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Mon, Oct 28, 2013 at 11:31:32AM +0100, Ingo Molnar wrote:
> 
> * Borislav Petkov <bp@alien8.de> wrote:
> 
> > On Mon, Oct 28, 2013 at 11:17:49AM +0100, Ingo Molnar wrote:
> >
> > > And regular write-back cacheable isn't sufficient because the 
> > > CPU could do things like prefetch your range automatically?
> > 
> > Yeah, he's doing a CLFLUSH anyway which basically makes it a 
> > write-through...
> 
> The CLFLUSH is done afterwards (making it a use-once thing), so WB 
> might still be faster and would avoid the PAT headache ...
> 
> Thanks,
> 
> 	Ingo
What i do right now is:
1. clflush the data range to read from my mmio device
2. read the data.
On PCIe Tracer i see the pcie  bursts.

If i mark the region WB and call clflush my system will crash without
any message, it just stop working.

regards
Andy

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Ingo Molnar]

* Andreas Werner <wernerandy@gmx.de> wrote:

> On Mon, Oct 28, 2013 at 11:31:32AM +0100, Ingo Molnar wrote:
> > 
> > * Borislav Petkov <bp@alien8.de> wrote:
> > 
> > > On Mon, Oct 28, 2013 at 11:17:49AM +0100, Ingo Molnar wrote:
> > >
> > > > And regular write-back cacheable isn't sufficient because the 
> > > > CPU could do things like prefetch your range automatically?
> > > 
> > > Yeah, he's doing a CLFLUSH anyway which basically makes it a 
> > > write-through...
> > 
> > The CLFLUSH is done afterwards (making it a use-once thing), so WB 
> > might still be faster and would avoid the PAT headache ...
> > 
> > Thanks,
> > 
> > 	Ingo
> What i do right now is:
> 1. clflush the data range to read from my mmio device
> 2. read the data.
> On PCIe Tracer i see the pcie  bursts.
> 
> If i mark the region WB and call clflush my system will crash without
> any message, it just stop working.

Yeah, I was wondering whether it's valid at all to mark IO memory as 
cacheable - with the lack of MESI transactions and all that ...

So it's apparently not valid and we've got to live with WT as the 
'best' caching/bursting method for reads.

Thanks,

	Ingo

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[H. Peter Anvin]

On 10/28/2013 03:51 AM, Ingo Molnar wrote:
>>
>> If i mark the region WB and call clflush my system will crash without
>> any message, it just stop working.
> 
> Yeah, I was wondering whether it's valid at all to mark IO memory as 
> cacheable - with the lack of MESI transactions and all that ...
> 
> So it's apparently not valid and we've got to live with WT as the 
> 'best' caching/bursting method for reads.
> 

On a lot of systems I'm not sure if even WT is legitimate.

	-hpa

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Borislav Petkov]

On Mon, Oct 28, 2013 at 11:34:28AM +0100, Andreas Werner wrote:
> Yes the reads are only for packet data, the commands or configuration
> registers are mapped non cachable.
> 
> I´ve tried WB, but on PCIe Tracer i could not see any burst access.
> Thats the reason why i have created this patch.
> 
> Is there a chance to get this patch into the kernel? Or
> is this solution so special?

Ok, but your patch returns WB pat type for WT MTRR type, AFAICT.

You want to do:

  PAT=Write-Back + MTRR=Write-Through = Effective Memory of Write-Through

but you end up doing

  PAT=Write-Back + MTRR=Write-Through = Effective Memory of Write-Back

What am I missing or misunderstanding?

AFAICT, you want to return _PAGE_PWT for MTRR_TYPE_WRTHROUGH, no?

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Mon, Oct 28, 2013 at 11:51:01AM +0100, Ingo Molnar wrote:
> 
> * Andreas Werner <wernerandy@gmx.de> wrote:
> 
> > On Mon, Oct 28, 2013 at 11:31:32AM +0100, Ingo Molnar wrote:
> > > 
> > > * Borislav Petkov <bp@alien8.de> wrote:
> > > 
> > > > On Mon, Oct 28, 2013 at 11:17:49AM +0100, Ingo Molnar wrote:
> > > >
> > > > > And regular write-back cacheable isn't sufficient because the 
> > > > > CPU could do things like prefetch your range automatically?
> > > > 
> > > > Yeah, he's doing a CLFLUSH anyway which basically makes it a 
> > > > write-through...
> > > 
> > > The CLFLUSH is done afterwards (making it a use-once thing), so WB 
> > > might still be faster and would avoid the PAT headache ...
> > > 
> > > Thanks,
> > > 
> > > 	Ingo
> > What i do right now is:
> > 1. clflush the data range to read from my mmio device
> > 2. read the data.
> > On PCIe Tracer i see the pcie  bursts.
> > 
> > If i mark the region WB and call clflush my system will crash without
> > any message, it just stop working.
> 
> Yeah, I was wondering whether it's valid at all to mark IO memory as 
> cacheable - with the lack of MESI transactions and all that ...
> 
> So it's apparently not valid and we've got to live with WT as the 
> 'best' caching/bursting method for reads.
> 
> Thanks,
> 
> 	Ingo

Yeah, as you can see in the link i´ve posted before, the guy who
did the post mentioned also that WB on MMIO is not valid, he said
"id could work on some CPUs", and therefore he decided to do it
like I with WC (write) and WT (read).

regards Andy

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Mon, Oct 28, 2013 at 11:57:31AM +0100, Borislav Petkov wrote:
> On Mon, Oct 28, 2013 at 11:34:28AM +0100, Andreas Werner wrote:
> > Yes the reads are only for packet data, the commands or configuration
> > registers are mapped non cachable.
> > 
> > I´ve tried WB, but on PCIe Tracer i could not see any burst access.
> > Thats the reason why i have created this patch.
> > 
> > Is there a chance to get this patch into the kernel? Or
> > is this solution so special?
> 
> Ok, but your patch returns WB pat type for WT MTRR type, AFAICT.
> 
> You want to do:
> 
>   PAT=Write-Back + MTRR=Write-Through = Effective Memory of Write-Through
>
Yes thats right.
 
> but you end up doing
> 
>   PAT=Write-Back + MTRR=Write-Through = Effective Memory of Write-Back
>
No the effective memory type is WT, check out the Intel document with the
table of Effective memory type combinations.
 
> What am I missing or misunderstanding?
> 
> AFAICT, you want to return _PAGE_PWT for MTRR_TYPE_WRTHROUGH, no?

Yes but, there is no way in the kernel to mark a memory WT,
there is just ioremap_wc for Write combining and ioremap_cache
for Write Back, and as you can see in the Intel Effective Memory type
table, if you combine PAT=WB and MTRR=WT you will get a effective memory
of WT.

regards 
Andy
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> Sent from a fat crate under my desk. Formatting is fine.
> --

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Borislav Petkov]

Drop Dave's stale mail address.

On Mon, Oct 28, 2013 at 12:25:05PM +0100, Andreas Werner wrote:
> > but you end up doing
> > 
> >   PAT=Write-Back + MTRR=Write-Through = Effective Memory of Write-Back
> >
> No the effective memory type is WT, check out the Intel document with the
> table of Effective memory type combinations.

You need to read what I'm saying more carefully.

> Yes but, there is no way in the kernel to mark a memory WT,

That doesn't mean you can return _PAGE_CACHE_WB for MTRR_TYPE_WRTHROUGH.
The correct thing to do, IMHO, would be to set the PWT bit in the PTEs
of those pages comprising your buffer.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Mon, Oct 28, 2013 at 12:45:41PM +0100, Borislav Petkov wrote:
> Drop Dave's stale mail address.
> 
> On Mon, Oct 28, 2013 at 12:25:05PM +0100, Andreas Werner wrote:
> > > but you end up doing
> > > 
> > >   PAT=Write-Back + MTRR=Write-Through = Effective Memory of Write-Back
> > >
> > No the effective memory type is WT, check out the Intel document with the
> > table of Effective memory type combinations.
> 
> You need to read what I'm saying more carefully.
Sorry then i missunderstood something?
> 
> > Yes but, there is no way in the kernel to mark a memory WT,
> 
> That doesn't mean you can return _PAGE_CACHE_WB for MTRR_TYPE_WRTHROUGH.
> The correct thing to do, IMHO, would be to set the PWT bit in the PTEs
> of those pages comprising your buffer.
> 
But why? the combination of PAT=WB and MTRR=WT is allowed isn´t it?
What should i return instead of?

regards
Andy
> -- 
> Regards/Gruss,
>     Boris.
> 
> Sent from a fat crate under my desk. Formatting is fine.
> --

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Borislav Petkov]

On Mon, Oct 28, 2013 at 01:03:58PM +0100, Andreas Werner wrote:
> > That doesn't mean you can return _PAGE_CACHE_WB for MTRR_TYPE_WRTHROUGH.
> > The correct thing to do, IMHO, would be to set the PWT bit in the PTEs
> > of those pages comprising your buffer.
> > 
> But why? the combination of PAT=WB and MTRR=WT is allowed isn´t it?
> What should i return instead of?

Well, since the pat code can't give you PWT, I'm thinking the cleaner
thing would be to do it yourself. I.e., allocate a WT buffer with
mtrr_add and then set the PWT bit of each page in that buffer.

Alternatively, your solution - albeit not that correct - is a simple one
and probably could use a good comment in the code explaining why you're
doing that if we decide to go that way.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

On Mon, Oct 28, 2013 at 02:58:47PM +0100, Borislav Petkov wrote:
> On Mon, Oct 28, 2013 at 01:03:58PM +0100, Andreas Werner wrote:
> > > That doesn't mean you can return _PAGE_CACHE_WB for MTRR_TYPE_WRTHROUGH.
> > > The correct thing to do, IMHO, would be to set the PWT bit in the PTEs
> > > of those pages comprising your buffer.
> > > 
> > But why? the combination of PAT=WB and MTRR=WT is allowed isn´t it?
> > What should i return instead of?
> 
> Well, since the pat code can't give you PWT, I'm thinking the cleaner
> thing would be to do it yourself. I.e., allocate a WT buffer with
> mtrr_add and then set the PWT bit of each page in that buffer.
> 
> Alternatively, your solution - albeit not that correct - is a simple one
> and probably could use a good comment in the code explaining why you're
> doing that if we decide to go that way.

At the moment if you check the function in pat.c there is also UC returned
if the requested type is not WB. So that was my idea to add WT and return WB.

If i understand it right, i have to add a comment why i do that
and if that is good enough, the patch could be submitted to the kernel?

regards 
Andy
 

> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> Sent from a fat crate under my desk. Formatting is fine.
> --

[PATCH] X86: MM: Add PAT Type write-through in combination with mtrr

[Andreas Werner]

This patch adds the Write-through memory type in combination with mtrr.
If you call ioremap_cache to request cachable memory (write-back) the function
tries to set the PAT to write-back only if the mtrr setting of the requested region
is also marked as Write-Back.

If the mttr regions are marked e.g. as Write-through or with other types, the function will
always return UC- memory.

If you check the Intel document " IA-32 SDM vol 3a table Effective Memory Type", there
are many other combinations possible.

This patch will only add the following combination:
  PAT=Write-Back + MTRR=Write-Through = Effective Memory of Write-Through

Tested on - Intel (R) Atom E680 (Tunnel Creek)
          - Intel (R) Core(TM)2 Duo

Signed-off-by: Andreas Werner <wernerandy@gmx.de>
---
 arch/x86/mm/pat.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/mm/pat.c b/arch/x86/mm/pat.c
index 6574388..9cfe107 100644
--- a/arch/x86/mm/pat.c
+++ b/arch/x86/mm/pat.c
@@ -149,10 +149,15 @@ static unsigned long pat_x_mtrr_type(u64 start, u64 end, unsigned long req_type)
 		u8 mtrr_type;
 
 		mtrr_type = mtrr_type_lookup(start, end);
-		if (mtrr_type != MTRR_TYPE_WRBACK)
-			return _PAGE_CACHE_UC_MINUS;
 
-		return _PAGE_CACHE_WB;
+		switch (mtrr_type) {
+		case MTRR_TYPE_WRBACK:
+		case MTRR_TYPE_WRTHROUGH:
+		  return _PAGE_CACHE_WB;
+
+		default:
+		  return _PAGE_CACHE_UC_MINUS;
+		}
 	}
 
 	return req_type;
-- 
1.8.3.4