Re: [PATCH v2] kni: add IOVA va support for kni

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 4/4/2019 10:57 AM, Burakov, Anatoly wrote:
> On 03-Apr-19 5:29 PM, Ferruh Yigit wrote:
>> On 4/1/2019 10:51 AM, Kiran Kumar Kokkilagadda wrote:
>>> From: Kiran Kumar K <kirankumark@marvell.com>
>>>
>>> With current KNI implementation kernel module will work only in
>>> IOVA=PA mode. This patch will add support for kernel module to work
>>> with IOVA=VA mode.
>>
>> Thanks Kiran for removing the limitation, I have a few questions, can you please
>> help me understand.
>>
>> And when this patch is ready, the restriction in 'linux/eal/eal.c', in
>> 'rte_eal_init' should be removed, perhaps with this patch. I assume you already
>> doing it to be able to test this patch.
>>
>>>
>>> The idea is to maintain a mapping in KNI module between user pages and
>>> kernel pages and in fast path perform a lookup in this table and get
>>> the kernel virtual address for corresponding user virtual address.
>>>
>>> In IOVA=VA mode, the memory allocated to the pool is physically
>>> and virtually contiguous. We will take advantage of this and create a
>>> mapping in the kernel.In kernel we need mapping for queues
>>> (tx_q, rx_q,... slow path) and mbuf memory (fast path).
>>
>> Is it?
>> As far as I know mempool can have multiple chunks and they can be both virtually
>> and physically separated.
>>
>> And even for a single chunk, that will be virtually continuous, but will it be
>> physically continuous?
> 
> Just to clarify.
> 
> Within DPDK, we do not make a distinction between physical address and 
> IOVA address - we never need the actual physical address, we just need 
> the DMA addresses, which can either match the physical address, or be 
> completely arbitrary (in our case, they will match VA addresses, but it 
> doesn't have to be the case - in fact, 17.11 will, under some 
> circumstances, populate IOVA addresses simply starting from address 0).
> 
> However, one has to remember that IOVA address *is not a physical 
> address*. The pages backing a VA chunk may be *IOVA*-contiguous, but may 
> not necessarily be *physically* contiguous. Under normal circumstances 
> we really don't care, because the VFIO/IOMMU takes care of the mapping 
> between IOVA and PA transparently for the hardware.
> 
> So, in IOVA as VA mode, the memory allocated to the mempool will be 
> (within a given chunk) both VA and IOVA contiguous - but not necessarily 
> *physically* contiguous! In fact, if you try calling rte_mem_virt2phy() 
> on the mempool pages, you'll likely find that they aren't (i've seen 
> cases where pages were mapped /backwards/!).
> 
> Therefore, unless by "physically contiguous" you mean "IOVA-contiguous", 
> you *cannot* rely on memory in mempool being *physically* contiguous 
> merely based on the fact that it's IOVA-contiguous.

Thanks for clarification.