DMA memory limitation?

DMA memory limitation?

[Vasu Varma P V]

Hi,

Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than
14MB of the mem using this on my PCI SMP box with 256MB ram.
I think there is restriction on ISA boards of 16MB.
Can we increase it ?

thx,
Vasu

Re: DMA memory limitation?

[Alan Cox]

> Is there any limitation on DMA memory we can allocate using
> kmalloc(size, GFP_DMA)? I am not able to acquire more than
> 14MB of the mem using this on my PCI SMP box with 256MB ram.
> I think there is restriction on ISA boards of 16MB.

Yes. GFP_DMA allocates memory below 16Mbytes for ISA devices and that is
a physical wiring issue. For PCI devices you can allocate a lot more using
the pci_alloc_* and pci_map_* interface to allocate memory for PCI in a
CPU independant manner. It'll also then look after cache coherency issues
for you

Re: DMA memory limitation?

[Arjan van de Ven]

Vasu Varma P V wrote:
> 
> Hi,
> 
> Is there any limitation on DMA memory we can allocate using
> kmalloc(size, GFP_DMA)? I am not able to acquire more than
> 14MB of the mem using this on my PCI SMP box with 256MB ram.
> I think there is restriction on ISA boards of 16MB.
> Can we increase it ?

Why?
YOu don't have to allocate GFP_DMA memory for PCI cards!
GFP_DMA is for ISA cards only

Re: DMA memory limitation?

[Vasu Varma P V]

but we have a macro in include/asm-i386/dma.h,
MAX_DMA_ADDRESS  (PAGE_OFFSET+0x1000000).

if i change it to a higher value, i am able to get more dma
memory. Is there any way i can change this without compiling
the kernel?

Arjan van de Ven wrote:

> Vasu Varma P V wrote:
> >
> > Hi,
> >
> > Is there any limitation on DMA memory we can allocate using
> > kmalloc(size, GFP_DMA)? I am not able to acquire more than
> > 14MB of the mem using this on my PCI SMP box with 256MB ram.
> > I think there is restriction on ISA boards of 16MB.
> > Can we increase it ?
>
> Why?
> YOu don't have to allocate GFP_DMA memory for PCI cards!
> GFP_DMA is for ISA cards only

Re: DMA memory limitation?

[Richard B. Johnson]

On Thu, 5 Jul 2001, Vasu Varma P V wrote:

> Hi,
> 
> Is there any limitation on DMA memory we can allocate using
> kmalloc(size, GFP_DMA)? I am not able to acquire more than
> 14MB of the mem using this on my PCI SMP box with 256MB ram.
> I think there is restriction on ISA boards of 16MB.
> Can we increase it ?
> 
> thx,
> Vasu

14MB of DMA(able) memory?  Err. I think you are trying to
do something you would never need to do.

If your board has a PCI interface, it's got address-space
allocated where memory does not exist. That's for communicating
with the board. If this address-space contains memory on that
board, you treat it just like RAM. That's what the PCI bus
does for you.

Given this, your board may also be a "bus mastering" board.
This means that, given the correct programming sequence, it
can transfer data to or from any location accessible to the
PCI bus. This is the so-called DMA that PCI bus masters can
perform.

When you program such a board, you simply translate the
virtual address of the transfer area to the address that
the board understands. Look at /linux/Documentation/IO-mapping.txt

The GFP_DMA memory type to which you refer, is for boards that
must use the lower 16 MB of address space. Such boards are
usually on the ISA bus. Since the first 1 MB is pretty much
taken by ROM BIOS, screen-card BIOS, other controllers (like SCSI)
BIOS, etc., you are not going to increase it.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

RE: DMA memory limitation?

[Christophe Beaumont]

> On Thu, 5 Jul 2001, Vasu Varma P V wrote:
> 
> > Hi,
> > 
> > Is there any limitation on DMA memory we can allocate using
> > kmalloc(size, GFP_DMA)? I am not able to acquire more than
> > 14MB of the mem using this on my PCI SMP box with 256MB ram.
> > I think there is restriction on ISA boards of 16MB.
> > Can we increase it ?
> > 
> > thx,
> > Vasu
> 
> 14MB of DMA(able) memory?  Err. I think you are trying to
> do something you would never need to do.
> 
And what is that supposed to be????

I have a piece of pretty well designed hardware here... and my
application requires me to have the PCI board to random access
in master mode a whole lot of memory (anywhere from 128MEGS to
1GIG... and possibly more...) so I really do need BIG DMA buffers
(I don't say huge anymore as one can get 1/2 Gig of RAM for just
over 120 bucks???)... 

There is no way I can have the piece of hardware behave in 
another fashion... and NO it is NOT broken (when you do BOTH 
hardware & software, you know about BOTH limitations... there
are just some cases where you have to face some unique issues).
the mem=1024M works pretty fine once you have figured out how 
to handle in a fairly simple way this "reserved" memory...

So please software people... do not blame it all on the 
hardware... =)

Chris.

RE: DMA memory limitation?

[Richard B. Johnson]

On Thu, 5 Jul 2001, Christophe Beaumont wrote:

> 
> 
> > On Thu, 5 Jul 2001, Vasu Varma P V wrote:
> > 
> > > Hi,
> > > 
> > > Is there any limitation on DMA memory we can allocate using
> > > kmalloc(size, GFP_DMA)? I am not able to acquire more than
> > > 14MB of the mem using this on my PCI SMP box with 256MB ram.
> > > I think there is restriction on ISA boards of 16MB.
> > > Can we increase it ?
> > > 
> > > thx,
> > > Vasu
> > 
> > 14MB of DMA(able) memory?  Err. I think you are trying to
> > do something you would never need to do.
> > 
> And what is that supposed to be????
> 

Did you READ the rest of the carefully-worded paragraphs,
carefully, explaining that it ** ISN'T DMA ** memory that you need?

> I have a piece of pretty well designed hardware here... and my
> application requires me to have the PCI board to random access
> in master mode a whole lot of memory (anywhere from 128MEGS to
     ^^^^^^^^^^^
You JUST give it the address of any RAM you want to read/write...
as explained before. As explained, in a documentation file I
specified, you convert the virtual address using a macro that
the kernel headers provide.

You can even give it user-mode buffers as long as they are non-paged
during the access.

> 1GIG... and possibly more...) so I really do need BIG DMA buffers
> (I don't say huge anymore as one can get 1/2 Gig of RAM for just
> over 120 bucks???)... 
> 
> There is no way I can have the piece of hardware behave in 
> another fashion... and NO it is NOT broken (when you do BOTH 
> hardware & software, you know about BOTH limitations... there
> are just some cases where you have to face some unique issues).
> the mem=1024M works pretty fine once you have figured out how 
> to handle in a fairly simple way this "reserved" memory...
> 

This is wrong! Good luck sucker.

> So please software people...
                     ^^^^^^^

I am a Software Engineer, with a long history of hardware design.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: DMA memory limitation?

[Helge Hafting]

Vasu Varma P V wrote:
> 
> Hi,
> 
> Is there any limitation on DMA memory we can allocate using
> kmalloc(size, GFP_DMA)? I am not able to acquire more than
> 14MB of the mem using this on my PCI SMP box with 256MB ram.
> I think there is restriction on ISA boards of 16MB.
> Can we increase it ?

You can allocate a lot more memory for your pci activities.
No problem there.  Just drop the "GFP_DMA" and you'll get 
up to 1G or so.

You shouldn't use GFP_DMA because PCI cards don't need that.
Only ISA cards needs GFP_DMA because they can't use more
than 16M.  So obviously GFP_DMA is limited to
16M because it is really ISA_DMA.

PCI don't need such special tricks, so don't use GFP_DMA!
Your PCI cards is able to DMA into any memory, including
the non-GFP_DMA memory.

> but we have a macro in include/asm-i386/dma.h,
> MAX_DMA_ADDRESS  (PAGE_OFFSET+0x1000000).
> 
> if i change it to a higher value, i am able to get more dma
> memory. Is there any way i can change this without compiling
> the kernel?
> 
No matter what you do, DON'T change that.  Yeah, you'll get
a bigger GFP_DMA pool, but that'll break each and every 
ISA card that tries to allocate GFP_DMA memory.  You
achieve exactly the same effect for your PCI card by ditching
the GFP_DMA parameter, but then you achieve it without breaking
ISA cards.

Helge Hafting

Re: DMA memory limitation?

[Steffen Persvold]

Helge Hafting wrote:
> 
> Vasu Varma P V wrote:
> >
> > Hi,
> >
> > Is there any limitation on DMA memory we can allocate using
> > kmalloc(size, GFP_DMA)? I am not able to acquire more than
> > 14MB of the mem using this on my PCI SMP box with 256MB ram.
> > I think there is restriction on ISA boards of 16MB.
> > Can we increase it ?
> 
> You can allocate a lot more memory for your pci activities.
> No problem there.  Just drop the "GFP_DMA" and you'll get
> up to 1G or so.
> 
> You shouldn't use GFP_DMA because PCI cards don't need that.
> Only ISA cards needs GFP_DMA because they can't use more
> than 16M.  So obviously GFP_DMA is limited to
> 16M because it is really ISA_DMA.
> 
> PCI don't need such special tricks, so don't use GFP_DMA!
> Your PCI cards is able to DMA into any memory, including
> the non-GFP_DMA memory.
> 
> > but we have a macro in include/asm-i386/dma.h,
> > MAX_DMA_ADDRESS  (PAGE_OFFSET+0x1000000).
> >
> > if i change it to a higher value, i am able to get more dma
> > memory. Is there any way i can change this without compiling
> > the kernel?
> >
> No matter what you do, DON'T change that.  Yeah, you'll get
> a bigger GFP_DMA pool, but that'll break each and every
> ISA card that tries to allocate GFP_DMA memory.  You
> achieve exactly the same effect for your PCI card by ditching
> the GFP_DMA parameter, but then you achieve it without breaking
> ISA cards.
> 
A problem arises on 64 bit platforms (such as IA64) if your PCI card is only 32bit (can
address the first 4G) and you don't wan't to use bounce buffers. If you use GFP_DMA on
IA64 you are ensured that the memory you get is below 4G and not 16M as on i386, hence no
bounce buffers are needed. On Alpha GFP_DMA is not limited at all (I think). Correct me if
I'm wrong, but I really think there should be a general way of allocating memory that is
32bit addressable (something like GFP_32BIT?) so you don't need a lot of #ifdef's in your
code.

Regards,
-- 
  Steffen Persvold               Systems Engineer
  Email : mailto:sp@scali.no     Scali AS (http://www.scali.com)
  Tlf   : (+47) 22 62 89 50      Olaf Helsets vei 6
  Fax   : (+47) 22 62 89 51      N-0621 Oslo, Norway

Re: DMA memory limitation?

[Alan Cox]

> A problem arises on 64 bit platforms (such as IA64) if your PCI card is only 32bit (can
> address the first 4G) and you don't wan't to use bounce buffers. If you use GFP_DMA on

GFP_DMA is ISA dma reachable, Forget the IA64, their setup is weird and 
should best be ignored until 2.5 as and when they sort it out.

> bounce buffers are needed. On Alpha GFP_DMA is not limited at all (I think). Correct me if

Alpha has various IOMMU facilities

> I'm wrong, but I really think there should be a general way of allocating memory that is
> 32bit addressable (something like GFP_32BIT?) so you don't need a lot of #ifdef's in your
> code.

No ifdefs are needed

	GFP_DMA - ISA dma reachable
	pci_alloc_* and friends - PCI usable memory

Re: DMA memory limitation?

[Jeff Garzik]

Alan Cox wrote:
> No ifdefs are needed
> 
>         GFP_DMA - ISA dma reachable
>         pci_alloc_* and friends - PCI usable memory

pci_alloc_* is designed to support ISA.

Pass pci_dev==NULL to pci_alloc_* for ISA devices, and it allocs GFP_DMA
for you.

-- 
Jeff Garzik      | A recent study has shown that too much soup
Building 1024    | can cause malaise in laboratory mice.
MandrakeSoft     |

Re: DMA memory limitation?

[Steffen Persvold]

> > GFP_DMA is ISA dma reachable, Forget the IA64, their setup is weird and 
> > should best be ignored until 2.5 as and when they sort it out.

Really ? I don't think I can ignore IA64, there are people who ask for it....

> > > bounce buffers are needed. On Alpha GFP_DMA is not limited at all (I think). Correct me if
> >
> > Alpha has various IOMMU facilities
> >
> > > I'm wrong, but I really think there should be a general way of allocating memory that is
> > > 32bit addressable (something like GFP_32BIT?) so you don't need a lot of #ifdef's in your
> > > code.
> > No ifdefs are needed
> >
> >         GFP_DMA - ISA dma reachable
> >         pci_alloc_* and friends - PCI usable memory
> 
> pci_alloc_* is designed to support ISA.
> 
> Pass pci_dev==NULL to pci_alloc_* for ISA devices, and it allocs GFP_DMA
> for you.
> 

Sure, but the IA64 platforms that are out now doesn't have an IOMMU, so bounce buffers are
used if you don't specify GFP_DMA in your get_free_page.

Now lets say you have a driver with a page allocator. Eventually you want to make some if
the allocated pages available to a 32bit PCI device. These pages has to be consistent (i.e
the driver doesn't have to wait for a PCI flush for the data to be valid, sort of like a
ethernet ring buffer). I could use the pci_alloc_consistent() function
(pci_alloc_consistent() allocates a buffer with GFP_DMA on IA64), but since I already have
the pages, I have to use pci_map_single (or pci_map_sg). Inside pci_map_single on IA64
something called swiotlb buffers (bounce buffers) are used if the device can't support
64bit addressing and the address of the memory to map is above the 4G limit. The swiotlb
buffers are below the 4G limit and therefore reachable by any PCI device. The problem
about these buffers are that the content are not copied to the original location before
you do a pci_sync_* or a pci_unmap_* (they are not consistent) and they are a limited
resource (allocated at boot time). My solution for now was to use :

#if defined(__ia64__)
  int flag = GFP_DMA;
#else
  int flag = 0;
#endif

Maybe IA64 could implement GFP_HIGHMEM (as on i386) so that if no flags were used you were
guaranteed to get 32bit memory ???


Regards
-- 
  Steffen Persvold               Systems Engineer
  Email : mailto:sp@scali.no     Scali AS (http://www.scali.com)
  Tlf   : (+47) 22 62 89 50      Olaf Helsets vei 6
  Fax   : (+47) 22 62 89 51      N-0621 Oslo, Norway

Re: DMA memory limitation?

[Jeff Garzik]

Steffen Persvold wrote:
> > pci_alloc_* is designed to support ISA.
> >
> > Pass pci_dev==NULL to pci_alloc_* for ISA devices, and it allocs GFP_DMA
> > for you.

> Sure, but the IA64 platforms that are out now doesn't have an IOMMU, so bounce buffers are
> used if you don't specify GFP_DMA in your get_free_page.
[...]
> (pci_alloc_consistent() allocates a buffer with GFP_DMA on IA64),

The important thing is that pci_alloc_consistent and the other PCI DMA
functions work as advertised on IA64.  If you pass NULL to
pci_alloc_consistent, IA64 should give you an ISA DMA-able address.  If
you don't, you get a 32-bit PCI DMA address.  Use of GFP_DMA is a
arch-specific detail, so don't let me confuse you there.

-- 
Jeff Garzik      | A recent study has shown that too much soup
Building 1024    | can cause malaise in laboratory mice.
MandrakeSoft     |

Re: DMA memory limitation?

[Alan Cox]

> > > GFP_DMA is ISA dma reachable, Forget the IA64, their setup is weird and 
> > > should best be ignored until 2.5 as and when they sort it out.
> 
> Really ? I don't think I can ignore IA64, there are people who ask for it....

Well the current IA64 tree isnt related to the standard Linux behaviour so you
need to take that up with the IA64 people. Obviously their non standard
behaviour causes problems.
> 

RE: DMA memory limitation?

[Matt_Domsch]

> The important thing is that pci_alloc_consistent and the other PCI DMA
> functions work as advertised on IA64.  If you pass NULL to
> pci_alloc_consistent, IA64 should give you an ISA DMA-able 
> address.  If
> you don't, you get a 32-bit PCI DMA address.  Use of GFP_DMA is a
> arch-specific detail, so don't let me confuse you there.

Until recently, on IA-64, pci_alloc_consistent() given a NULL pci_dev would
fault.  It's been fixed in at least the most recent IA-64 patch.
pci_map_single() and pci_map_sg() still have the same problem, as they
dereference pci_dev w/o checking for NULL first.


-- 
Matt Domsch
Sr. Software Engineer
Dell Linux Solutions
www.dell.com/linux
#2 Linux Server provider with 17% in the US and 14% Worldwide (IDC)!
#3 Unix provider with 18% in the US (Dataquest)!