[PATCH] eal: force IOVA PA mode if KNI module inserted

[PATCH] eal: force IOVA PA mode if KNI module inserted

[Ferruh Yigit]

Fix kernel crash with KNI because KNI requires physical addresses.

When IOVA VA mode used, memzones and mbufs physical address fields
contain virtual addresses. But KNI relies on these fields to enable
kernel access for buffers. Those fields having virtual address cause
crash in kernel.

This is a workaround until KNI fixed properly to work with virtual
addresses.

Fixes: 72d013644bd6 ("mem: honor IOVA mode in malloc virt2phy")

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Jianfeng Tan <jianfeng.tan@intel.com>
Cc: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Cc: Santosh Shukla <santosh.shukla@caviumnetworks.com>
Cc: Thomas Monjalon <thomas@monjalon.net>

This patch superseded following one:
http://dpdk.org/dev/patchwork/patch/31071/
---
 lib/librte_eal/linuxapp/eal/eal.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 017c402ed..29912a4e5 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -808,6 +808,15 @@ rte_eal_init(int argc, char **argv)
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
 
+	/* Workaround for KNI which requires physical address to work */
+	if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
+			rte_eal_check_module("rte_kni") == 1) {
+		rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+		RTE_LOG(WARNING, EAL,
+			"Some devices want iova as va but pa will be used because.. "
+			"KNI module inserted\n");
+	}
+
 	if (internal_config.no_hugetlbfs == 0 &&
 			internal_config.process_type != RTE_PROC_SECONDARY &&
 			eal_hugepage_info_init() < 0) {
-- 
2.13.6

Re: [PATCH] eal: force IOVA PA mode if KNI module inserted

[santosh]

Hi Ferruh,


On Thursday 02 November 2017 05:36 AM, Ferruh Yigit wrote:
> Fix kernel crash with KNI because KNI requires physical addresses.
>
> When IOVA VA mode used, memzones and mbufs physical address fields
> contain virtual addresses. But KNI relies on these fields to enable
> kernel access for buffers. Those fields having virtual address cause
> crash in kernel.
>
> This is a workaround until KNI fixed properly to work with virtual
> addresses.
>
> Fixes: 72d013644bd6 ("mem: honor IOVA mode in malloc virt2phy")
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: Jianfeng Tan <jianfeng.tan@intel.com>
> Cc: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Cc: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
>
> This patch superseded following one:
> http://dpdk.org/dev/patchwork/patch/31071/
> ---
>  lib/librte_eal/linuxapp/eal/eal.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
> index 017c402ed..29912a4e5 100644
> --- a/lib/librte_eal/linuxapp/eal/eal.c
> +++ b/lib/librte_eal/linuxapp/eal/eal.c
> @@ -808,6 +808,15 @@ rte_eal_init(int argc, char **argv)
>  	/* autodetect the iova mapping mode (default is iova_pa) */
>  	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
>  
> +	/* Workaround for KNI which requires physical address to work */
> +	if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
> +			rte_eal_check_module("rte_kni") == 1) {
> +		rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
> +		RTE_LOG(WARNING, EAL,
> +			"Some devices want iova as va but pa will be used because.. "
> +			"KNI module inserted\n");
> +	}
> +
>  	if (internal_config.no_hugetlbfs == 0 &&
>  			internal_config.process_type != RTE_PROC_SECONDARY &&
>  			eal_hugepage_info_init() < 0) {

Rather checking for KNI module at linuxapp/eal, I was suggesting to move
KNI detection code in bus layer like below:

diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index cdf8106..971586c 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -586,11 +586,13 @@ rte_pci_get_iommu_class(void)
        bool is_vfio_noiommu_enabled = true;
        bool has_iova_va;
        bool is_bound_uio;
+       bool has_kni;
 
        is_bound = pci_one_device_is_bound();
        if (!is_bound)
                return RTE_IOVA_DC;
 
+       has_kni = rte_eal_check_module("rte_kni");
        has_iova_va = pci_one_device_has_iova_va();
        is_bound_uio = pci_one_device_bound_uio();
 #ifdef VFIO_PRESENT
@@ -598,7 +600,7 @@ rte_pci_get_iommu_class(void)
                                        true : false;
 #endif
 
-       if (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled)
+       if (has_iova_va && !is_bound_uio && !has_kni && !is_vfio_noiommu_enabled)
                return RTE_IOVA_VA;
 
        if (has_iova_va) {
@@ -607,6 +609,9 @@ rte_pci_get_iommu_class(void)
                        RTE_LOG(WARNING, EAL, "vfio-noiommu mode configured\n");
                if (is_bound_uio)
                        RTE_LOG(WARNING, EAL, "few device bound to UIO\n");
+               if (has_kni)
+                       RTE_LOG(WARNING, EAL, "Some devices want iova as va but pa will be used because.."
+                                       "KNI module inserted\n");
        }
 
        return RTE_IOVA_PA;
It builds but I haven;t tested, can you please give it a try.

Thanks.

Re: [PATCH] eal: force IOVA PA mode if KNI module inserted

[Ferruh Yigit]

On 11/1/2017 8:58 PM, santosh wrote:
> Hi Ferruh,
> 
> 
> On Thursday 02 November 2017 05:36 AM, Ferruh Yigit wrote:
>> Fix kernel crash with KNI because KNI requires physical addresses.
>>
>> When IOVA VA mode used, memzones and mbufs physical address fields
>> contain virtual addresses. But KNI relies on these fields to enable
>> kernel access for buffers. Those fields having virtual address cause
>> crash in kernel.
>>
>> This is a workaround until KNI fixed properly to work with virtual
>> addresses.
>>
>> Fixes: 72d013644bd6 ("mem: honor IOVA mode in malloc virt2phy")
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: Jianfeng Tan <jianfeng.tan@intel.com>
>> Cc: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>> Cc: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>
>> This patch superseded following one:
>> http://dpdk.org/dev/patchwork/patch/31071/
>> ---
>>  lib/librte_eal/linuxapp/eal/eal.c | 9 +++++++++
>>  1 file changed, 9 insertions(+)
>>
>> diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
>> index 017c402ed..29912a4e5 100644
>> --- a/lib/librte_eal/linuxapp/eal/eal.c
>> +++ b/lib/librte_eal/linuxapp/eal/eal.c
>> @@ -808,6 +808,15 @@ rte_eal_init(int argc, char **argv)
>>  	/* autodetect the iova mapping mode (default is iova_pa) */
>>  	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
>>  
>> +	/* Workaround for KNI which requires physical address to work */
>> +	if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
>> +			rte_eal_check_module("rte_kni") == 1) {
>> +		rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
>> +		RTE_LOG(WARNING, EAL,
>> +			"Some devices want iova as va but pa will be used because.. "
>> +			"KNI module inserted\n");
>> +	}
>> +
>>  	if (internal_config.no_hugetlbfs == 0 &&
>>  			internal_config.process_type != RTE_PROC_SECONDARY &&
>>  			eal_hugepage_info_init() < 0) {
> 
> Rather checking for KNI module at linuxapp/eal, I was suggesting to move
> KNI detection code in bus layer like below:

I was thinking same at first, but later was not sure about add this check into
bus related file, kni module check is so unrelated with pci.c .

Also this is not part of a regular process of detecting iova mode, which putting
into rte_pci_get_iommu_class() will look like it is. Instead this is a
workaround and we are overwriting the result of regular method for this special
case, so I think putting this into linux eal.c level after detection is good for
stressing the workaround.

So lets not spoil the logic in rte_pci_get_iommu_class() with kni module checks.

> 
> diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
> index cdf8106..971586c 100644
> --- a/drivers/bus/pci/linux/pci.c
> +++ b/drivers/bus/pci/linux/pci.c
> @@ -586,11 +586,13 @@ rte_pci_get_iommu_class(void)
>         bool is_vfio_noiommu_enabled = true;
>         bool has_iova_va;
>         bool is_bound_uio;
> +       bool has_kni;
>  
>         is_bound = pci_one_device_is_bound();
>         if (!is_bound)
>                 return RTE_IOVA_DC;
>  
> +       has_kni = rte_eal_check_module("rte_kni");
>         has_iova_va = pci_one_device_has_iova_va();
>         is_bound_uio = pci_one_device_bound_uio();
>  #ifdef VFIO_PRESENT
> @@ -598,7 +600,7 @@ rte_pci_get_iommu_class(void)
>                                         true : false;
>  #endif
>  
> -       if (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled)
> +       if (has_iova_va && !is_bound_uio && !has_kni && !is_vfio_noiommu_enabled)
>                 return RTE_IOVA_VA;
>  
>         if (has_iova_va) {
> @@ -607,6 +609,9 @@ rte_pci_get_iommu_class(void)
>                         RTE_LOG(WARNING, EAL, "vfio-noiommu mode configured\n");
>                 if (is_bound_uio)
>                         RTE_LOG(WARNING, EAL, "few device bound to UIO\n");
> +               if (has_kni)
> +                       RTE_LOG(WARNING, EAL, "Some devices want iova as va but pa will be used because.."
> +                                       "KNI module inserted\n");
>         }
>  
>         return RTE_IOVA_PA;
> It builds but I haven;t tested, can you please give it a try.
> 
> Thanks.
> 

Re: [PATCH] eal: force IOVA PA mode if KNI module inserted

[Thomas Monjalon]

02/11/2017 01:06, Ferruh Yigit:
> Fix kernel crash with KNI because KNI requires physical addresses.
> 
> When IOVA VA mode used, memzones and mbufs physical address fields
> contain virtual addresses. But KNI relies on these fields to enable
> kernel access for buffers. Those fields having virtual address cause
> crash in kernel.
> 
> This is a workaround until KNI fixed properly to work with virtual
> addresses.
> 
> Fixes: 72d013644bd6 ("mem: honor IOVA mode in malloc virt2phy")
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>

Applied, thanks