PCIe dma-able memory location in physical memory for Intel Nehalem + Tylersburg architecture

PCIe dma-able memory location in physical memory for Intel Nehalem + Tylersburg architecture

[Rob van de Voort]

Hello,

I have the following question regarding the (re-)allocation of dma-able
memory for a PCIe device. I wish to know on which CPU / Memory
controller the dma-able memory will be allocated. In addition i would
like to know how i can control this allocation process, or how to
reroute it to another CPU / Memory controller.

The system i use contains two Nehalem processors connected via intel QPI
bus. Both processors have 3GB of DDR3 RAM in their memory banks. The
PCIe bus is controlled by a tylersburg io hub which is also connected to
the intel QPI bus

The system runs a 64bit linux kernel. Furthermore a PCIe device which
can only use 32 bit addressing is connected to the PCIe bus. I want to
allocate dma-able memory for this device using a driver, i know how to
do this. For maximal performance i would like to control the allocation
of dma-able memory, in particular the allocation to a certain CPU /
memory controller. The tylersburg IO hub has an influence on this
allocation but can i influence this.

So my questions are:
-1- How can i know were our dma-able memory is allocated in physical
memory?
-2- How can i control / redirect this allocation to a certain memory
controller / CPU?

Thanks for any reply,

Rob van de Voort

P.S  I'd be grateful if people could point me in the direction of
resources I could read. Thus far I have gone over the PCI / DMA
documentation in
the kernel, and a couple of articles on lwn. 
If this is the wrong mailing list for this type of question i hope you
can redirect me to the correct one.



Disclaimer: The information contained in this email, including any attachments is 
confidential and is for the sole use of the intended recipient(s). Any unauthorized 
review, use, disclosure or distribution is prohibited. If you are not the intended 
recipient, please notify the sender immediately by replying to this message and 
destroy all copies of this message and any attachments.

Re: PCIe dma-able memory location in physical memory for Intel Nehalem + Tylersburg architecture

[Robert Hancock]

On 07/09/2009 03:04 AM, Rob van de Voort wrote:
> Hello,
>
> I have the following question regarding the (re-)allocation of dma-able
> memory for a PCIe device. I wish to know on which CPU / Memory
> controller the dma-able memory will be allocated. In addition i would
> like to know how i can control this allocation process, or how to
> reroute it to another CPU / Memory controller.
>
> The system i use contains two Nehalem processors connected via intel QPI
> bus. Both processors have 3GB of DDR3 RAM in their memory banks. The
> PCIe bus is controlled by a tylersburg io hub which is also connected to
> the intel QPI bus
>
> The system runs a 64bit linux kernel. Furthermore a PCIe device which
> can only use 32 bit addressing is connected to the PCIe bus. I want to
> allocate dma-able memory for this device using a driver, i know how to
> do this. For maximal performance i would like to control the allocation
> of dma-able memory, in particular the allocation to a certain CPU /
> memory controller. The tylersburg IO hub has an influence on this
> allocation but can i influence this.
>
> So my questions are:
> -1- How can i know were our dma-able memory is allocated in physical
> memory?
> -2- How can i control / redirect this allocation to a certain memory
> controller / CPU?
>
> Thanks for any reply,
>
> Rob van de Voort
>
> P.S  I'd be grateful if people could point me in the direction of
> resources I could read. Thus far I have gone over the PCI / DMA
> documentation in
> the kernel, and a couple of articles on lwn.
> If this is the wrong mailing list for this type of question i hope you
> can redirect me to the correct one.

I believe that dma_alloc_coherent, etc. normally try to allocate DMA 
memory on the same node as the device is connected to. For that to 
actually happen, the motherboard's ACPI tables have to be set up to 
indicate the proper node to memory allocations. As well, since your 
device can only address 32-bit memory (below 4GB) this may prevent 
memory that gets allocated from being on the optimal CPU.

Re: PCIe dma-able memory location in physical memory for Intel Nehalem + Tylersburg architecture

[Andi Kleen]

On Fri, Jul 10, 2009 at 09:56:29AM +0200, Rob van de Voort wrote:
> 
> Hello,
> 
> I have the following question regarding the (re-)allocation of dma-able
> memory for a PCIe device. I wish to know on which CPU / Memory
> controller the dma-able memory will be allocated. 

There are several kinds of dma-able memory.

Coherent/consistent memory (pci_alloc_consistent, dma_alloc_coherent)
This is usually allocated on the node where the device is connected to,
if the platform reports it correctly using ACPI and there is enough
memory available on that node.  In some kernels there were bugs with this.

All other memory, later mapped with pci_map_*/dma_map_*: This 
is allocated according to the current numa memory policy, unless
you override it (alloc_pages_node etc.)

> The system runs a 64bit linux kernel. Furthermore a PCIe device which
> can only use 32 bit addressing is connected to the PCIe bus. I want to

GFP_DMA32 allocations are typically only on the first node, unless
your first node has less than 3GB of memory or so.

The only thing you can do for 4GB limited allocation devices is to
enable VT-d and allocate it elsewhere (alloc_pages_node et.al.)
and then let it be remapped by pci_map*/dma_map* into the first
4GB.

> P.S  I'd be grateful if people could point me in the direction of
> resources I could read. Thus far I have gone over the PCI / DMA

http://halobates.de/numaapi3.pdf has a general overview of NUMA
policy, although from the user space perspective.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

PCIe dma-able memory location in physical memory for Intel Nehalem + Tylersburg architecture

[Rob van de Voort]

Hello,

I have the following question regarding the (re-)allocation of dma-able
memory for a PCIe device. I wish to know on which CPU / Memory
controller the dma-able memory will be allocated. In addition i would
like to know how i can control this allocation process, or how to
reroute it to another CPU / Memory controller.

The system i use contains two Nehalem processors connected via intel QPI
bus. Both processors have 3GB of DDR3 RAM in their memory banks. The
PCIe bus is controlled by a tylersburg io hub which is also connected to
the intel QPI bus

The system runs a 64bit linux kernel. Furthermore a PCIe device which
can only use 32 bit addressing is connected to the PCIe bus. I want to
allocate dma-able memory for this device using a driver, i know how to
do this. For maximal performance i would like to control the allocation
of dma-able memory, in particular the allocation to a certain CPU /
memory controller. The tylersburg IO hub has an influence on this
allocation but can i influence this.

So my questions are:
-1- How can i know were our dma-able memory is allocated in physical
memory?
-2- How can i control / redirect this allocation to a certain memory
controller / CPU?

Thanks for any reply,

Rob van de Voort

P.S  I'd be grateful if people could point me in the direction of
resources I could read. Thus far I have gone over the PCI / DMA
documentation in the kernel, and a couple of articles on lwn. 
If this is the wrong mailing list for this type of question i hope you
can redirect me to the correct one.

Disclaimer: The information contained in this email, including any attachments is 
confidential and is for the sole use of the intended recipient(s). Any unauthorized 
review, use, disclosure or distribution is prohibited. If you are not the intended 
recipient, please notify the sender immediately by replying to this message and 
destroy all copies of this message and any attachments.