clean way to support >32bit addr on 32bit CPU

clean way to support >32bit addr on 32bit CPU

[Dave]

I have this ARM (XScale) based platform that supports 36bit physical
addressing. Due to the way the ATU is designed, the outbound memory
translation window is fixed outside the first 4GB of memory space, and
thus the need to use 64bit addressing in order to access the PCI bus. 
After all said and done, the struct resource members start and end
must support 64bit integer values in order to work. On a 64bit arch
that would be fine since unsigned long is 64bit. However on a 32bit
arch one must use unsigned long long to get 64bit. However, if we do
that then it would make the 64bit archs to have 128bit start and end
and probably wouldn't be something we'd want. What would be a nice
clean way to support this that's acceptable to Linux? I guess this
issue would be similar to x86-32 PAE would have?

Also, please cc me on on the discussion. Not sure if my LKML
subscription is working... Thanks!

-- 
-= Dave =-

Software Engineer - Advanced Development Engineering Team 
Storage Component Division - Intel Corp. 
mailto://dave.jiang @ intel dot com
http://sourceforge.net/projects/xscaleiop/
----
The views expressed in this email are
mine alone and do not necessarily 
reflect the views of my employer
(Intel Corp.).

Re: clean way to support >32bit addr on 32bit CPU

[Deepak Saxena]

On Jan 10 2005, at 16:01, Slade Maurer was caught saying:
> Also, it would be nice to have PTEs to represent the upper 4GB such that 
> it can be mmapped to user space. PAE handled this in and it would be 
> great to have it in ARM MMU36 as well.

Not doable. I believe PAE allows for normal 4K pages to be used when
mapping > 32-bits. XSC3 and ARMv6 only allow for > 32 bit addresses 
when using 16MB pages (supersections), so we need to instead use
the hugetlb approach.

~Deepak

-- 
Deepak Saxena - dsaxena@plexity.net - http://www.plexity.net

If you complain once more, you'll meet an army of me. - Bjork

Re: clean way to support >32bit addr on 32bit CPU

[Slade Maurer]

Dave wrote:

>I have this ARM (XScale) based platform that supports 36bit physical
>addressing. Due to the way the ATU is designed, the outbound memory
>translation window is fixed outside the first 4GB of memory space, and
>thus the need to use 64bit addressing in order to access the PCI bus. 
>After all said and done, the struct resource members start and end
>must support 64bit integer values in order to work. On a 64bit arch
>that would be fine since unsigned long is 64bit. However on a 32bit
>arch one must use unsigned long long to get 64bit. However, if we do
>that then it would make the 64bit archs to have 128bit start and end
>and probably wouldn't be something we'd want. What would be a nice
>clean way to support this that's acceptable to Linux? I guess this
>issue would be similar to x86-32 PAE would have?
>
>Also, please cc me on on the discussion. Not sure if my LKML
>subscription is working... Thanks!
>
>  
>
Also, it would be nice to have PTEs to represent the upper 4GB such that 
it can be mmapped to user space. PAE handled this in and it would be 
great to have it in ARM MMU36 as well.

  -Slade

Re: clean way to support >32bit addr on 32bit CPU

[Roland Dreier]

    Dave> I have this ARM (XScale) based platform that supports 36bit
    Dave> physical addressing. Due to the way the ATU is designed, the
    Dave> outbound memory translation window is fixed outside the
    Dave> first 4GB of memory space, and thus the need to use 64bit
    Dave> addressing in order to access the PCI bus. After all said
    Dave> and done, the struct resource members start and end must
    Dave> support 64bit integer values in order to work. On a 64bit
    Dave> arch that would be fine since unsigned long is
    Dave> 64bit. However on a 32bit arch one must use unsigned long
    Dave> long to get 64bit. However, if we do that then it would make
    Dave> the 64bit archs to have 128bit start and end and probably
    Dave> wouldn't be something we'd want. What would be a nice clean
    Dave> way to support this that's acceptable to Linux? I guess this
    Dave> issue would be similar to x86-32 PAE would have?

Actually unsigned long long is still 64 bits even on 64-bit Linux
architectures.  However it might make more sense to use an explicit
size for the resource addresses, namely u64.

This XScale architecture seems similar to the PowerPC 440, which also
uses 36-bit addressing for various buses including PCI.  You might
want to take a look at arch/ppc for inspiration.

 - Roland

Re: clean way to support >32bit addr on 32bit CPU

[Linus Torvalds]

On Mon, 10 Jan 2005, Dave wrote:
>
> After all said and done, the struct resource members start and end
> must support 64bit integer values in order to work. On a 64bit arch
> that would be fine since unsigned long is 64bit. However on a 32bit
> arch one must use unsigned long long to get 64bit.

We really should make "struct resource" use u64's. It's wrong even on x86, 
but we've never seen any real problems in practice, so we've had little 
reason to bother.

This has definitely come up before, maybe there's even some old patch 
floating around. It should be as easy as just fixing up "start/end" to be 
"u64" (and perhaps move them to the beginning of the struct to make sure 
packing is ok on all architectures), and fixing any fall-out.

		Linus

Re: clean way to support >32bit addr on 32bit CPU

[Randy.Dunlap]

Linus Torvalds wrote:
> 
> On Mon, 10 Jan 2005, Dave wrote:
> 
>>After all said and done, the struct resource members start and end
>>must support 64bit integer values in order to work. On a 64bit arch
>>that would be fine since unsigned long is 64bit. However on a 32bit
>>arch one must use unsigned long long to get 64bit.
> 
> 
> We really should make "struct resource" use u64's. It's wrong even on x86, 
> but we've never seen any real problems in practice, so we've had little 
> reason to bother.
> 
> This has definitely come up before, maybe there's even some old patch 
> floating around. It should be as easy as just fixing up "start/end" to be 
> "u64" (and perhaps move them to the beginning of the struct to make sure 
> packing is ok on all architectures), and fixing any fall-out.

Speaking of fall-out, or more like trickle-down,
I'm almost done with a patch to make PCMCIA resources use
unsigned long instead of u_int or u_short for IO address:

incluce/pcmcia/cs_types.h:
#if defined(__arm__) || defined(__mips__)
typedef u_int   ioaddr_t;
#else
typedef u_short	ioaddr_t;
#endif

becomes:
typedef unsigned long	ioaddr_t;

and then include/pcmcia/cs.c needs some changes in use of
ioaddr_t, along with drivers (printk formats).

Does that sound OK?
I guess that it would become unsigned long long (or u64)
with this proposal?

-- 
~Randy

Re: clean way to support >32bit addr on 32bit CPU

[Slade Maurer]

Deepak Saxena wrote:

>On Jan 10 2005, at 16:01, Slade Maurer was caught saying:
>  
>
>>Also, it would be nice to have PTEs to represent the upper 4GB such that 
>>it can be mmapped to user space. PAE handled this in and it would be 
>>great to have it in ARM MMU36 as well.
>>    
>>
>
>Not doable. I believe PAE allows for normal 4K pages to be used when
>mapping > 32-bits. XSC3 and ARMv6 only allow for > 32 bit addresses 
>when using 16MB pages (supersections), so we need to instead use
>the hugetlb approach.
>
>~Deepak
>
>  
>
You are right of course. The MMUs first level descriptors force you to 
have 16MB pages.

I don't see anything wrong with using hugeTLB. Then it is up to the user 
to get hugetlbfs setup so that they can mmap(...) properly. This is 
forced on us by the designers of the MMU ;)

I think that is better than setting permissions during ioremap(...) so 
that a user space process can use a kernel virtual address for user 
space access.

 -Slade

Re: clean way to support >32bit addr on 32bit CPU

[Linus Torvalds]

On Mon, 10 Jan 2005, Randy.Dunlap wrote:
> 
> Speaking of fall-out, or more like trickle-down,
> I'm almost done with a patch to make PCMCIA resources use
> unsigned long instead of u_int or u_short for IO address:

Ahh, yes. That's required on pretty much all platforms except x86 and
x86-64.

Of course, since ARM and MIPS already do the "u_int" thing, and not a 
whole lot of other architectures do PCMCIA, I guess it doesn't matter 
_that_ much. Cardbus stuff should get it right regardless.

> typedef unsigned long	ioaddr_t;
> 
> and then include/pcmcia/cs.c needs some changes in use of
> ioaddr_t, along with drivers (printk formats).
> 
> Does that sound OK?
> I guess that it would become unsigned long long (or u64)
> with this proposal?

I don't think ioaddr_t needs to match resources. None of the IO accessor
functions take "u64"s anyway - and aren't likely to do so in the future
either - so "unsigned long" should be good enough.

Having u64 for resource handling is mainly an issue for RAM and
memory-mapped IO (right now the 32-bit limit means that we throw away
information about stuff above the 4GB mark from the e820 interfaces on
x86, for example - that _happens_ to work because we never see anything 
but RAM there anyway, but it means that /proc/iomem doesn't show all of 
the system RAM, and it does mean that our resource management doesn't 
actually handle 64-bit addresses correctly. 

See drivers/pci/probe.c for the result:

	"PCI: Unable to handle 64-bit address for device xxxx"

(and I do not actually think this has _ever_ happened in real life, which 
makes me suspect that Windows doesn't handle them either - but it 
inevitably will happen some day).

		Linus

Re: clean way to support >32bit addr on 32bit CPU

[William Lee Irwin III]

On Mon, Jan 10, 2005 at 05:30:25PM -0800, Linus Torvalds wrote:
> I don't think ioaddr_t needs to match resources. None of the IO accessor
> functions take "u64"s anyway - and aren't likely to do so in the future
> either - so "unsigned long" should be good enough.
> Having u64 for resource handling is mainly an issue for RAM and
> memory-mapped IO (right now the 32-bit limit means that we throw away
> information about stuff above the 4GB mark from the e820 interfaces on
> x86, for example - that _happens_ to work because we never see anything 
> but RAM there anyway, but it means that /proc/iomem doesn't show all of 
> the system RAM, and it does mean that our resource management doesn't 
> actually handle 64-bit addresses correctly. 
> See drivers/pci/probe.c for the result:
> 	"PCI: Unable to handle 64-bit address for device xxxx"
> (and I do not actually think this has _ever_ happened in real life, which 
> makes me suspect that Windows doesn't handle them either - but it 
> inevitably will happen some day).

I have a vague recollection of seeing a report of an ia32 device and/or
machine with this property from John Fusco but am having a tough time
searching the archives properly for it. I do recall it being around the
time the remap_pfn_range() work was started, and I also claimed it as
one of the motivators of it in one of my posts. I'm unaware of whether
there are more general resources in John Fusco's situation.

My follow-ups began with:
Message-ID: <20040924021735.GL9106@holomorphy.com>
References: <41535AAE.6090700@yahoo.com>



-- wli

Re: clean way to support >32bit addr on 32bit CPU

[Randy.Dunlap]

William Lee Irwin III wrote:
> On Mon, Jan 10, 2005 at 05:30:25PM -0800, Linus Torvalds wrote:
> 
>>I don't think ioaddr_t needs to match resources. None of the IO accessor
>>functions take "u64"s anyway - and aren't likely to do so in the future
>>either - so "unsigned long" should be good enough.
>>Having u64 for resource handling is mainly an issue for RAM and
>>memory-mapped IO (right now the 32-bit limit means that we throw away
>>information about stuff above the 4GB mark from the e820 interfaces on
>>x86, for example - that _happens_ to work because we never see anything 
>>but RAM there anyway, but it means that /proc/iomem doesn't show all of 
>>the system RAM, and it does mean that our resource management doesn't 
>>actually handle 64-bit addresses correctly. 
>>See drivers/pci/probe.c for the result:
>>	"PCI: Unable to handle 64-bit address for device xxxx"
>>(and I do not actually think this has _ever_ happened in real life, which 
>>makes me suspect that Windows doesn't handle them either - but it 
>>inevitably will happen some day).
> 
> 
> I have a vague recollection of seeing a report of an ia32 device and/or
> machine with this property from John Fusco but am having a tough time
> searching the archives properly for it. I do recall it being around the
> time the remap_pfn_range() work was started, and I also claimed it as
> one of the motivators of it in one of my posts. I'm unaware of whether
> there are more general resources in John Fusco's situation.
> 
> My follow-ups began with:
> Message-ID: <20040924021735.GL9106@holomorphy.com>
> References: <41535AAE.6090700@yahoo.com>

http://marc.theaimsgroup.com/?l=linux-mm&m=109598180125156&w=2

-- 
~Randy

Re: clean way to support >32bit addr on 32bit CPU

[Randy.Dunlap]

Linus Torvalds wrote:
> 
> On Mon, 10 Jan 2005, Randy.Dunlap wrote:
> 
>>Speaking of fall-out, or more like trickle-down,
>>I'm almost done with a patch to make PCMCIA resources use
>>unsigned long instead of u_int or u_short for IO address:
> 
> Ahh, yes. That's required on pretty much all platforms except x86 and
> x86-64.

OK, I don't get it, sorry.  What's different about ARM & MIPS here
(for PCMCIA)?  Is this historical (so that I'm just missing it)
or is it a data types difference?

> Of course, since ARM and MIPS already do the "u_int" thing, and not a 
> whole lot of other architectures do PCMCIA, I guess it doesn't matter 
> _that_ much. Cardbus stuff should get it right regardless.
> 
> 
>>typedef unsigned long	ioaddr_t;
>>
>>and then include/pcmcia/cs.c needs some changes in use of
>>ioaddr_t, along with drivers (printk formats).
>>
>>Does that sound OK?
>>I guess that it would become unsigned long long (or u64)
>>with this proposal?
> 
> 
> I don't think ioaddr_t needs to match resources. None of the IO accessor
> functions take "u64"s anyway - and aren't likely to do so in the future
> either - so "unsigned long" should be good enough.

Thanks,
-- 
~Randy

Re: clean way to support >32bit addr on 32bit CPU

[Linus Torvalds]

On Tue, 11 Jan 2005, Randy.Dunlap wrote:
>
> > Ahh, yes. That's required on pretty much all platforms except x86 and
> > x86-64.
> 
> OK, I don't get it, sorry.  What's different about ARM & MIPS here
> (for PCMCIA)?  Is this historical (so that I'm just missing it)
> or is it a data types difference?

Nothing is different. Pretty much every architecture - except for x86 and
ilk - will at least have the _potential_ for IO ports encoded above the
16-bit mark.

But a lot of architectures won't have PCMCIA (or if they do, they end up
having the whole ISA mapping, and for compatibility reasons they'll end up
making the ports visible to the kernel in the low 16 bits, even if the
actual hw has some other physical translation - I think that's true on
ppc, at least).

So what makes ARM and MIPS special is just the fact that they have PCMCIA, 
but don't necessarily have the traditional ISA mappings. Embedded devices 
and all that. Others either try to hide the fact that they look different, 
or just never cared.

But the right thing is definitely to make an IO port pointer be "unsigned 
int" or even "unsigned long". 

		Linus

Re: clean way to support >32bit addr on 32bit CPU

[Dave]

On Mon, 10 Jan 2005 16:09:57 -0800 (PST), Linus Torvalds
<torvalds@osdl.org> wrote:
> 
> 
> On Mon, 10 Jan 2005, Dave wrote:
> >
> > After all said and done, the struct resource members start and end
> > must support 64bit integer values in order to work. On a 64bit arch
> > that would be fine since unsigned long is 64bit. However on a 32bit
> > arch one must use unsigned long long to get 64bit.
> 
> We really should make "struct resource" use u64's. It's wrong even on x86,
> but we've never seen any real problems in practice, so we've had little
> reason to bother.
> 
> This has definitely come up before, maybe there's even some old patch
> floating around. It should be as easy as just fixing up "start/end" to be
> "u64" (and perhaps move them to the beginning of the struct to make sure
> packing is ok on all architectures), and fixing any fall-out.
> 
>                 Linus
> 

Shall I change the PCI resource stuff also to be u64 or leave that
alone?  Currently I think we assume on 32bit processors all PCI
resources are in the first 4GB region. Just wondering if that should
be left alone....

-- 
-= Dave =-

Software Engineer - Advanced Development Engineering Team 
Storage Component Division - Intel Corp. 
mailto://dave.jiang @ intel
http://sourceforge.net/projects/xscaleiop/
----
The views expressed in this email are
mine alone and do not necessarily 
reflect the views of my employer
(Intel Corp.).