UIO: uio_mem does not handle devices above 4 GB address

UIO: uio_mem does not handle devices above 4 GB address

[Herrera-Bendezu, Luis]

I am working on a PPC440EPx board with some FPGAs located at
addresses above 4 GB and kernel version 2.6.28.7. The UIO driver is
used to access the FPGAs. I get following error when trying to read
FPGA registers:
Machine check in kernel mode.
Data Read PLB Error

The problem is traced to member addr in struct uio_mem:
struct uio_mem {
    unsigned long     addr;
    unsigned long     size;
    int               memtype;
    void __iomem      *internal_addr;
    struct uio_map    *map;
};

When UIO_MEM_PHYS is used with device address > 32-bit then
uio_mmap_physical() remaps address using idev->info->mem[mi].addr
thus loosing high address bits.

One way to solve this problem is to change addr member to:
    phys_addr_t       addr;

and modify corresponding attributes in uio.c. Only concern is that
uio_vma_fault() calls virt_to_page() and vmalloc_to_page() where both
have addr arg (unsigned long) and (const void *) respectively for
powerpc. But these functions are only used for UIO_MEM_LOGICAL and
UIO_MEM_VIRTUAL.

Any alternative approach?

Thanks,
Luis

Re: UIO: uio_mem does not handle devices above 4 GB address

[Hans J. Koch]

On Wed, Apr 01, 2009 at 01:20:49PM -0400, Herrera-Bendezu, Luis wrote:
> I am working on a PPC440EPx board with some FPGAs located at
> addresses above 4 GB

That's not exactly a good idea on a 32-bit processor. Can't you remap
that chip select to a decent address?

> and kernel version 2.6.28.7. The UIO driver is
> used to access the FPGAs. I get following error when trying to read
> FPGA registers:
> Machine check in kernel mode.
> Data Read PLB Error
> 
> The problem is traced to member addr in struct uio_mem:
> struct uio_mem {
>     unsigned long     addr;
>     unsigned long     size;
>     int               memtype;
>     void __iomem      *internal_addr;
>     struct uio_map    *map;
> };
> 
> When UIO_MEM_PHYS is used with device address > 32-bit then
> uio_mmap_physical() remaps address using idev->info->mem[mi].addr
> thus loosing high address bits.
> 
> One way to solve this problem is to change addr member to:
>     phys_addr_t       addr;

That alone won't solve the problem.

> 
> and modify corresponding attributes in uio.c. Only concern is that
> uio_vma_fault() calls virt_to_page() and vmalloc_to_page() where both
> have addr arg (unsigned long) and (const void *) respectively for
> powerpc. But these functions are only used for UIO_MEM_LOGICAL and
> UIO_MEM_VIRTUAL.

That probably won't work without large modifications.

> 
> Any alternative approach?

Yes, fix your hardware design.

Thanks,
Hans

RE: UIO: uio_mem does not handle devices above 4 GB address

[Herrera-Bendezu, Luis]

Hans, 

-----Original Message-----
From: Hans J. Koch [mailto:hjk@linutronix.de] 
Sent: Wednesday, April 01, 2009 4:11 PM
To: Herrera-Bendezu, Luis
Cc: linux-kernel@vger.kernel.org
Subject: Re: UIO: uio_mem does not handle devices above 4 GB address


On Wed, Apr 01, 2009 at 01:20:49PM -0400, Herrera-Bendezu, Luis wrote:
>> I am working on a PPC440EPx board with some FPGAs located at
>> addresses above 4 GB

> That's not exactly a good idea on a 32-bit processor. Can't you remap
> that chip select to a decent address?
Well PPC440EPx implements a 32-bit effective address (EA) space. EA
is expanded into virtual address and then translated to 33-bit (8 GB)
real address by the MMU so real addresses > 4 GB are valid and used by
this CPU.

>> and kernel version 2.6.28.7. The UIO driver is
>> used to access the FPGAs. I get following error when trying to read
>> FPGA registers:
>> Machine check in kernel mode.
>> Data Read PLB Error
>> 
>> The problem is traced to member addr in struct uio_mem:
>> struct uio_mem {
>>     unsigned long     addr;
>>     unsigned long     size;
>>     int               memtype;
>>     void __iomem      *internal_addr;
>>     struct uio_map    *map;
>> };
>> 
>> When UIO_MEM_PHYS is used with device address > 32-bit then
>> uio_mmap_physical() remaps address using idev->info->mem[mi].addr
>> thus loosing high address bits.
>> 
>> One way to solve this problem is to change addr member to:
>>     phys_addr_t       addr;

> That alone won't solve the problem.

>> 
>> and modify corresponding attributes in uio.c. Only concern is that
>> uio_vma_fault() calls virt_to_page() and vmalloc_to_page() where both
>> have addr arg (unsigned long) and (const void *) respectively for
>> powerpc. But these functions are only used for UIO_MEM_LOGICAL and
>> UIO_MEM_VIRTUAL.

> That probably won't work without large modifications.
Can you elaborate a little more on the mods needed?

>> 
>> Any alternative approach?

> Yes, fix your hardware design.
As described above this CPU can to address up to 8 GB.
>
> Thanks,
> Hans

Thanks,
Luis

Re: UIO: uio_mem does not handle devices above 4 GB address

[Hans J. Koch]

On Wed, Apr 01, 2009 at 05:01:23PM -0400, Herrera-Bendezu, Luis wrote:
> Hans, 
> 
> -----Original Message-----
> From: Hans J. Koch [mailto:hjk@linutronix.de] 
> Sent: Wednesday, April 01, 2009 4:11 PM
> To: Herrera-Bendezu, Luis
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: UIO: uio_mem does not handle devices above 4 GB address
> 
> 
> On Wed, Apr 01, 2009 at 01:20:49PM -0400, Herrera-Bendezu, Luis wrote:
> >> I am working on a PPC440EPx board with some FPGAs located at
> >> addresses above 4 GB
> 
> > That's not exactly a good idea on a 32-bit processor. Can't you remap
> > that chip select to a decent address?
> Well PPC440EPx implements a 32-bit effective address (EA) space. EA
> is expanded into virtual address and then translated to 33-bit (8 GB)
> real address by the MMU so real addresses > 4 GB are valid and used by
> this CPU.

That doesn't mean it's advisable to put hardware there. Some hardware
explicitly forbids it. Try installing a PCI card above 4GB...

> 
> >> and kernel version 2.6.28.7. The UIO driver is
> >> used to access the FPGAs. I get following error when trying to read
> >> FPGA registers:
> >> Machine check in kernel mode.
> >> Data Read PLB Error
> >> 
> >> The problem is traced to member addr in struct uio_mem:
> >> struct uio_mem {
> >>     unsigned long     addr;
> >>     unsigned long     size;
> >>     int               memtype;
> >>     void __iomem      *internal_addr;
> >>     struct uio_map    *map;
> >> };
> >> 
> >> When UIO_MEM_PHYS is used with device address > 32-bit then
> >> uio_mmap_physical() remaps address using idev->info->mem[mi].addr
> >> thus loosing high address bits.
> >> 
> >> One way to solve this problem is to change addr member to:
> >>     phys_addr_t       addr;
> 
> > That alone won't solve the problem.
> 
> >> 
> >> and modify corresponding attributes in uio.c. Only concern is that
> >> uio_vma_fault() calls virt_to_page() and vmalloc_to_page() where both
> >> have addr arg (unsigned long) and (const void *) respectively for
> >> powerpc. But these functions are only used for UIO_MEM_LOGICAL and
> >> UIO_MEM_VIRTUAL.
> 
> > That probably won't work without large modifications.
> Can you elaborate a little more on the mods needed?

I guess you'd have to look at the whole memory management stuff of each
architecture to find out which kind of memory can be mapped with addresses
above unsigned long. Hardware often needs more than one contigous pages.
It might well be possible that a certain arch could have RAM for user virtual
addresses above 4GB, but no hardware. I don't know PPC well enough to say
anything about its behaviour.

Thanks,
Hans

Re: UIO: uio_mem does not handle devices above 4 GB address

[Benjamin Herrenschmidt]

On Thu, 2009-04-02 at 09:20 -0400, Steven A. Falco wrote:

> struct uio_mem {
> 	phys_addr_t		addr;
> 	phys_addr_t		size;
> 	int			memtype;
> 	void __iomem		*internal_addr;
> 	struct uio_map		*map;
> };
> 
> A few other changes would be needed.  We'd have to use something
> other than

Use a struct resource instead.

> 	return sprintf(buf, "0x%lx\n", mem->addr);
> 

The struct resource has a printk format string.

Cheers,
Ben.
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev

Re: UIO: uio_mem does not handle devices above 4 GB address

[Benjamin Herrenschmidt]

On Thu, 2009-04-02 at 09:20 -0400, Steven A. Falco wrote:

> struct uio_mem {
> 	phys_addr_t		addr;
> 	phys_addr_t		size;
> 	int			memtype;
> 	void __iomem		*internal_addr;
> 	struct uio_map		*map;
> };
> 
> A few other changes would be needed.  We'd have to use something
> other than

Use a struct resource instead.

> 	return sprintf(buf, "0x%lx\n", mem->addr);
> 

The struct resource has a printk format string.

Cheers,
Ben.
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev

Re: UIO: uio_mem does not handle devices above 4 GB address

[Steven A. Falco]

> On Wed, Apr 01, 2009 at 01:20:49PM -0400, Herrera-Bendezu, Luis wrote:
> >> I am working on a PPC440EPx board with some FPGAs located at
> >> addresses above 4 GB

Cc'ing the PPC list, since they have the most experience with this
CPU, and may have some useful insights.  The topic is "UIO on the
PPC440EPx - 36-bit physical addresses".

The PPC440EPx uses 36-bit addressing but is a 32-bit processor.
*All* peripherals are above the first 4 GB - that cannot be changed
if you use this CPU chip.

In general, the solution is to use "struct resource" to carry physical
addresses and sizes.  The basic type is:

#ifdef CONFIG_PHYS_ADDR_T_64BIT
typedef u64 phys_addr_t;
#else
typedef u32 phys_addr_t;
#endif

and for this processor, we use u64 rather than u32.

So, if we want to make UIO usable on this class of CPU, we might
replace:

struct uio_mem {
	unsigned long		addr;
	unsigned long		size;
	int			memtype;
	void __iomem		*internal_addr;
	struct uio_map		*map;
};

with:

struct uio_mem {
	phys_addr_t		addr;
	phys_addr_t		size;
	int			memtype;
	void __iomem		*internal_addr;
	struct uio_map		*map;
};

A few other changes would be needed.  We'd have to use something
other than

	return sprintf(buf, "0x%lx\n", mem->addr);

because the format would be incorrect for a u64.  Also, I believe
we would get warnings in uio_vma_fault().  However, the phys_addr
is really only applicable to UIO_MEM_PHYS, so perhaps we want a
union of phys_addr_t and unsigned long for addr and size?

Anyway, this is a real problem for the PPC440EPx, that board 
designers cannot work around.

	Steve