[dpdk-dev] [PATCH v3 0/3] add travis ci support for ppc64le

[dpdk-dev] [PATCH v3 0/3] add travis ci support for ppc64le

[David Wilder]

This patch series adds Travis gcc compilation jobs for ppc64le.
Limitations for ppc64le are similar to arm64 (see commit
31bb45bcfd).

1. Only gcc builds are supported on ppc64le.

2. A container is used, therefor Huge pages are not available.
   Unit tests are not run, a single test (test-null) using the
   no-huge option is run.

V2: Insures iova-mode is VA when --no-huge is selected.
    Removed setting of --iova-mode=VA in test-null.sh.
V3: Refactor ppc64le changes on top of David Marchand's
    Patch series: "Reorganise Travis jobs"
    http://mails.dpdk.org/archives/dev/2020-February/158231.html

A Travis build with theses patches can be found here:
https://travis-ci.org/djlwilder/dpdk/builds/653196974

David Wilder (3):
  eal/linux: select iova-mode va with no-huge option
  devtools: allow test-null.sh to run on ppc64le
  ci: add travis ci support for native ppc64le

 .travis.yml                    | 20 ++++++++++++++++++++
 devtools/test-null.sh          |  2 +-
 lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)

-- 
2.25.0

[dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option

[David Wilder]

If --no-huge is set and iova-mode has not been specified force VA mode.
If --no-huge and --iova-mode=PA is requested error out as this is
an impossible configuration.

Signed-off-by: David Wilder <dwilder@us.ibm.com>
---
 lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
index 9530ee55f..d3a0a1731 100644
--- a/lib/librte_eal/linux/eal/eal.c
+++ b/lib/librte_eal/linux/eal/eal.c
@@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv)
 
 	/* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme */
 	if (internal_config.iova_mode == RTE_IOVA_DC) {
+
 		/* autodetect the IOVA mapping mode */
 		enum rte_iova_mode iova_mode = rte_bus_get_iommu_class();
 
+		if (iova_mode == RTE_IOVA_PA && !rte_eal_has_hugepages()) {
+			iova_mode = RTE_IOVA_VA;
+			RTE_LOG(WARNING, EAL, "Some buses want 'PA' but forcing 'VA' because --no-huge is requested.\n");
+			RTE_LOG(WARNING, EAL, "Not all buses may be able to initialize.\n");
+		}
+
 		if (iova_mode == RTE_IOVA_DC) {
 			RTE_LOG(DEBUG, EAL, "Buses did not request a specific IOVA mode.\n");
 
@@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv)
 			internal_config.iova_mode;
 	}
 
+	if (rte_eal_iova_mode() == RTE_IOVA_PA &&
+	    rte_eal_has_hugepages() == 0) {
+		rte_eal_init_alert("Cannot use IOVA as 'PA' with --no-huge");
+		rte_errno = EINVAL;
+		return -1;
+	}
+
 	if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) {
 		rte_eal_init_alert("Cannot use IOVA as 'PA' since physical addresses are not available");
 		rte_errno = EINVAL;
-- 
2.25.0

[dpdk-dev] [PATCH v3 2/3] devtools: allow test-null.sh to run on ppc64le

[David Wilder]

Memory requirements are larger for ppc64le
due to a higher RTE_MAX_LCORE value.

Signed-off-by: David Wilder <dwilder@us.ibm.com>
Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
---
 devtools/test-null.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/devtools/test-null.sh b/devtools/test-null.sh
index 548de8113..b7defe1fd 100755
--- a/devtools/test-null.sh
+++ b/devtools/test-null.sh
@@ -27,6 +27,6 @@ else
 fi
 
 (sleep 1 && echo stop) |
-$testpmd -c $coremask --no-huge -m 20 \
+$testpmd -c $coremask --no-huge -m 30 \
 	$libs -w 0:0.0 --vdev net_null1 --vdev net_null2 $eal_options -- \
 	--no-mlockall --total-num-mbufs=2048 $testpmd_options -ia
-- 
2.25.0

[dpdk-dev] [PATCH v3 3/3] ci: add travis ci support for native ppc64le

[David Wilder]

This change follows the example of aarch64 Travis support
by adding support for ppc64le gcc builds.  Limitations for
ppc64le are the same as aarch64 as described in:
commit 31bb45bcfdf5 ("ci: add travis ci support for native aarch64").

Signed-off-by: David Wilder <dwilder@us.ibm.com>
Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
---
 .travis.yml | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index b64a81bd0..ff0c48de1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -124,3 +124,23 @@ jobs:
   - env: DEF_LIB="shared"
     arch: arm64
     compiler: clang
+  # ppc64le gcc jobs
+  - env: DEF_LIB="static"
+    arch: ppc64le
+    compiler: gcc
+  - env: DEF_LIB="shared" BUILD_DOCS=1
+    arch: ppc64le
+    compiler: gcc
+    addons:
+      apt:
+        packages:
+          - *required_packages
+          - *doc_packages
+  - env: DEF_LIB="shared" ABI_CHECKS=1
+    arch: ppc64le
+    compiler: gcc
+    addons:
+      apt:
+        packages:
+          - *required_packages
+          - *libabigail_build_packages
-- 
2.25.0

Re: [dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option

[Jerin Jacob]

On Fri, Feb 21, 2020 at 4:22 AM David Wilder <dwilder@us.ibm.com> wrote:
>
> If --no-huge is set and iova-mode has not been specified force VA mode.
> If --no-huge and --iova-mode=PA is requested error out as this is
> an impossible configuration.
>
> Signed-off-by: David Wilder <dwilder@us.ibm.com>
> ---
>  lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
> index 9530ee55f..d3a0a1731 100644
> --- a/lib/librte_eal/linux/eal/eal.c
> +++ b/lib/librte_eal/linux/eal/eal.c
> @@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv)
>
>         /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme */
>         if (internal_config.iova_mode == RTE_IOVA_DC) {
> +
>                 /* autodetect the IOVA mapping mode */
>                 enum rte_iova_mode iova_mode = rte_bus_get_iommu_class();
>
> +               if (iova_mode == RTE_IOVA_PA && !rte_eal_has_hugepages()) {
> +                       iova_mode = RTE_IOVA_VA;

What if igb_uio or vfio_nommu has been loaded(i.e no iommu support
enabled from the driver)? This would fail.

> +                       RTE_LOG(WARNING, EAL, "Some buses want 'PA' but forcing 'VA' because --no-huge is requested.\n");
> +                       RTE_LOG(WARNING, EAL, "Not all buses may be able to initialize.\n");
> +               }
> +
>                 if (iova_mode == RTE_IOVA_DC) {
>                         RTE_LOG(DEBUG, EAL, "Buses did not request a specific IOVA mode.\n");
>
> @@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv)
>                         internal_config.iova_mode;
>         }
>
> +       if (rte_eal_iova_mode() == RTE_IOVA_PA &&
> +           rte_eal_has_hugepages() == 0) {
> +               rte_eal_init_alert("Cannot use IOVA as 'PA' with --no-huge");

Top of the tree already detecting this case. am I missing anything?

[master]dell[dpdk.org] $ sudo ./build/app/test/dpdk-test  -c 0x3
--no-huge --iova-mode=pa
EAL: Detected 56 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Static memory layout is selected, amount of reserved memory can
be adjusted with -m or --socket-mem
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available

> +               rte_errno = EINVAL;
> +               return -1;
> +       }
> +
>         if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) {
>                 rte_eal_init_alert("Cannot use IOVA as 'PA' since physical addresses are not available");
>                 rte_errno = EINVAL;
> --
> 2.25.0
>

Re: [dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option

[dwilder]

Thanks you for your review Jerin.  See my responses are inline.

On 2020-03-20 06:24, Jerin Jacob wrote:
> On Fri, Feb 21, 2020 at 4:22 AM David Wilder <dwilder@us.ibm.com> 
> wrote:
>> 
>> If --no-huge is set and iova-mode has not been specified force VA 
>> mode.
>> If --no-huge and --iova-mode=PA is requested error out as this is
>> an impossible configuration.
>> 
>> Signed-off-by: David Wilder <dwilder@us.ibm.com>
>> ---
>>  lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++
>>  1 file changed, 14 insertions(+)
>> 
>> diff --git a/lib/librte_eal/linux/eal/eal.c 
>> b/lib/librte_eal/linux/eal/eal.c
>> index 9530ee55f..d3a0a1731 100644
>> --- a/lib/librte_eal/linux/eal/eal.c
>> +++ b/lib/librte_eal/linux/eal/eal.c
>> @@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv)
>> 
>>         /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme 
>> */
>>         if (internal_config.iova_mode == RTE_IOVA_DC) {
>> +
>>                 /* autodetect the IOVA mapping mode */
>>                 enum rte_iova_mode iova_mode = 
>> rte_bus_get_iommu_class();
>> 
>> +               if (iova_mode == RTE_IOVA_PA && 
>> !rte_eal_has_hugepages()) {
>> +                       iova_mode = RTE_IOVA_VA;

> 
> What if igb_uio or vfio_nommu has been loaded(i.e no iommu support
> enabled from the driver)? This would fail.

Yes they would fail.  If igb_uio or vfio_nommu (or any driver) cant be 
forced to VA mode it cant be used with out hugepages.  Drivers can be 
available but not used therefor we print a warning message.

> 
>> +                       RTE_LOG(WARNING, EAL, "Some buses want 'PA' 
>> but forcing 'VA' because --no-huge is requested.\n");
>> +                       RTE_LOG(WARNING, EAL, "Not all buses may be 
>> able to initialize.\n");
>> +               }
>> +
>>                 if (iova_mode == RTE_IOVA_DC) {
>>                         RTE_LOG(DEBUG, EAL, "Buses did not request a 
>> specific IOVA mode.\n");
>> 
>> @@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv)
>>                         internal_config.iova_mode;
>>         }
>> 
>> +       if (rte_eal_iova_mode() == RTE_IOVA_PA &&
>> +           rte_eal_has_hugepages() == 0) {
>> +               rte_eal_init_alert("Cannot use IOVA as 'PA' with 
>> --no-huge");
> 
> Top of the tree already detecting this case. am I missing anything?
> 
> [master]dell[dpdk.org] $ sudo ./build/app/test/dpdk-test  -c 0x3
> --no-huge --iova-mode=pa
> EAL: Detected 56 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Static memory layout is selected, amount of reserved memory can
> be adjusted with -m or --socket-mem
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not 
> available
> EAL: Cannot use IOVA as 'PA' since physical addresses are not available
> 

The check you reference is reporting that physical address are not 
available, for example no permissions to read /proc/self/pagemap.  In 
this case, if --no-huge is set then PA mode is not allowed. There is no 
guarantee that physical address are persistent with out using hugepages.


>> +               rte_errno = EINVAL;
>> +               return -1;
>> +       }
>> +
>>         if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) {
>>                 rte_eal_init_alert("Cannot use IOVA as 'PA' since 
>> physical addresses are not available");
>>                 rte_errno = EINVAL;
>> --
>> 2.25.0
>> 

Re: [dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option

[Jerin Jacob]

On Mon, Mar 23, 2020 at 11:11 PM dwilder <dwilder@us.ibm.com> wrote:
>
> Thanks you for your review Jerin.  See my responses are inline.
>
> On 2020-03-20 06:24, Jerin Jacob wrote:
> > On Fri, Feb 21, 2020 at 4:22 AM David Wilder <dwilder@us.ibm.com>
> > wrote:
> >>
> >> If --no-huge is set and iova-mode has not been specified force VA
> >> mode.
> >> If --no-huge and --iova-mode=PA is requested error out as this is
> >> an impossible configuration.
> >>
> >> Signed-off-by: David Wilder <dwilder@us.ibm.com>
> >> ---
> >>  lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++
> >>  1 file changed, 14 insertions(+)
> >>
> >> diff --git a/lib/librte_eal/linux/eal/eal.c
> >> b/lib/librte_eal/linux/eal/eal.c
> >> index 9530ee55f..d3a0a1731 100644
> >> --- a/lib/librte_eal/linux/eal/eal.c
> >> +++ b/lib/librte_eal/linux/eal/eal.c
> >> @@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv)
> >>
> >>         /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme
> >> */
> >>         if (internal_config.iova_mode == RTE_IOVA_DC) {
> >> +
> >>                 /* autodetect the IOVA mapping mode */
> >>                 enum rte_iova_mode iova_mode =
> >> rte_bus_get_iommu_class();
> >>
> >> +               if (iova_mode == RTE_IOVA_PA &&
> >> !rte_eal_has_hugepages()) {
> >> +                       iova_mode = RTE_IOVA_VA;
>
> >
> > What if igb_uio or vfio_nommu has been loaded(i.e no iommu support
> > enabled from the driver)? This would fail.
>
> Yes they would fail.  If igb_uio or vfio_nommu (or any driver) cant be
> forced to VA mode it cant be used with out hugepages.  Drivers can be
> available but not used therefor we print a warning message.

I think, the warning will not be enough as the system will fail anyway.

iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == 1
case, we need to return error.

iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == 0
case warning is enough.

>
> >
> >> +                       RTE_LOG(WARNING, EAL, "Some buses want 'PA'
> >> but forcing 'VA' because --no-huge is requested.\n");
> >> +                       RTE_LOG(WARNING, EAL, "Not all buses may be
> >> able to initialize.\n");
> >> +               }
> >> +
> >>                 if (iova_mode == RTE_IOVA_DC) {
> >>                         RTE_LOG(DEBUG, EAL, "Buses did not request a
> >> specific IOVA mode.\n");
> >>
> >> @@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv)
> >>                         internal_config.iova_mode;
> >>         }
> >>
> >> +       if (rte_eal_iova_mode() == RTE_IOVA_PA &&
> >> +           rte_eal_has_hugepages() == 0) {
> >> +               rte_eal_init_alert("Cannot use IOVA as 'PA' with
> >> --no-huge");
> >
> > Top of the tree already detecting this case. am I missing anything?
> >
> > [master]dell[dpdk.org] $ sudo ./build/app/test/dpdk-test  -c 0x3
> > --no-huge --iova-mode=pa
> > EAL: Detected 56 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Static memory layout is selected, amount of reserved memory can
> > be adjusted with -m or --socket-mem
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not
> > available
> > EAL: Cannot use IOVA as 'PA' since physical addresses are not available
> >
>
> The check you reference is reporting that physical address are not
> available, for example no permissions to read /proc/self/pagemap.  In
> this case, if --no-huge is set then PA mode is not allowed. There is no
> guarantee that physical address are persistent with out using hugepages.

Since this check is under the following, Yes, make sense for the check.
The old command has explicit  --iova-mode=pa. So it is in the
different code paths.

/* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme */
        if (internal_config.iova_mode == RTE_IOVA_DC) {

>
>
> >> +               rte_errno = EINVAL;
> >> +               return -1;
> >> +       }
> >> +
> >>         if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) {
> >>                 rte_eal_init_alert("Cannot use IOVA as 'PA' since
> >> physical addresses are not available");
> >>                 rte_errno = EINVAL;
> >> --
> >> 2.25.0
> >>

Re: [dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option

[dwilder]

On 2020-03-23 23:19, Jerin Jacob wrote:
> On Mon, Mar 23, 2020 at 11:11 PM dwilder <dwilder@us.ibm.com> wrote:
>> 
>> Thanks you for your review Jerin.  See my responses are inline.
>> 
>> On 2020-03-20 06:24, Jerin Jacob wrote:
>> > On Fri, Feb 21, 2020 at 4:22 AM David Wilder <dwilder@us.ibm.com>
>> > wrote:
>> >>
>> >> If --no-huge is set and iova-mode has not been specified force VA
>> >> mode.
>> >> If --no-huge and --iova-mode=PA is requested error out as this is
>> >> an impossible configuration.
>> >>
>> >> Signed-off-by: David Wilder <dwilder@us.ibm.com>
>> >> ---
>> >>  lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++
>> >>  1 file changed, 14 insertions(+)
>> >>
>> >> diff --git a/lib/librte_eal/linux/eal/eal.c
>> >> b/lib/librte_eal/linux/eal/eal.c
>> >> index 9530ee55f..d3a0a1731 100644
>> >> --- a/lib/librte_eal/linux/eal/eal.c
>> >> +++ b/lib/librte_eal/linux/eal/eal.c
>> >> @@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv)
>> >>
>> >>         /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme
>> >> */
>> >>         if (internal_config.iova_mode == RTE_IOVA_DC) {
>> >> +
>> >>                 /* autodetect the IOVA mapping mode */
>> >>                 enum rte_iova_mode iova_mode =
>> >> rte_bus_get_iommu_class();
>> >>
>> >> +               if (iova_mode == RTE_IOVA_PA &&
>> >> !rte_eal_has_hugepages()) {
>> >> +                       iova_mode = RTE_IOVA_VA;
>> 
>> >
>> > What if igb_uio or vfio_nommu has been loaded(i.e no iommu support
>> > enabled from the driver)? This would fail.
>> 
>> Yes they would fail.  If igb_uio or vfio_nommu (or any driver) cant be
>> forced to VA mode it cant be used with out hugepages.  Drivers can be
>> available but not used therefor we print a warning message.
> 
> I think, the warning will not be enough as the system will fail anyway.
> 
> iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == 
> 1
> case, we need to return error.
> 
> iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == 
> 0
> case warning is enough.

The current code will skip the bus if iova-mode is not supported,  this 
allow other devices to continu on.  The handing of an unsupported 
iova-mode is done in rte_pci_probe_one_driver(). See also 
rte_bus_get_iommu_class() if multiple busses cant agree on iova-mode a 
warning is given.

Here I have bound 0002:01:00.1 to igb_uio and forced iova-mode=pa,  much 
as my code did when --no-huge is used.

./dpdk-devbind.py -s
0002:01:00.1 'Ethernet Controller X710/X557-AT 10GBASE-T 1589' 
drv=igb_uio unused=i40e
<....>
dpdk-testpmd -c 3  --iova-mode=va -w 0002:01:00.1 -- -ia
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: PCI device 0002:01:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1589 net_i40e
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not 
initializing  <<<<<<<<<
EAL: Requested device 0002:01:00.1 cannot be used
<.....>


>> 
>> >
>> >> +                       RTE_LOG(WARNING, EAL, "Some buses want 'PA'
>> >> but forcing 'VA' because --no-huge is requested.\n");
>> >> +                       RTE_LOG(WARNING, EAL, "Not all buses may be
>> >> able to initialize.\n");
>> >> +               }
>> >> +
>> >>                 if (iova_mode == RTE_IOVA_DC) {
>> >>                         RTE_LOG(DEBUG, EAL, "Buses did not request a
>> >> specific IOVA mode.\n");
>> >>
>> >> @@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv)
>> >>                         internal_config.iova_mode;
>> >>         }
>> >>
>> >> +       if (rte_eal_iova_mode() == RTE_IOVA_PA &&
>> >> +           rte_eal_has_hugepages() == 0) {
>> >> +               rte_eal_init_alert("Cannot use IOVA as 'PA' with
>> >> --no-huge");
>> >
>> > Top of the tree already detecting this case. am I missing anything?
>> >
>> > [master]dell[dpdk.org] $ sudo ./build/app/test/dpdk-test  -c 0x3
>> > --no-huge --iova-mode=pa
>> > EAL: Detected 56 lcore(s)
>> > EAL: Detected 2 NUMA nodes
>> > EAL: Static memory layout is selected, amount of reserved memory can
>> > be adjusted with -m or --socket-mem
>> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> > EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not
>> > available
>> > EAL: Cannot use IOVA as 'PA' since physical addresses are not available
>> >
>> 
>> The check you reference is reporting that physical address are not
>> available, for example no permissions to read /proc/self/pagemap.  In
>> this case, if --no-huge is set then PA mode is not allowed. There is 
>> no
>> guarantee that physical address are persistent with out using 
>> hugepages.
> 
> Since this check is under the following, Yes, make sense for the check.
> The old command has explicit  --iova-mode=pa. So it is in the
> different code paths.
> 
> /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme */
>         if (internal_config.iova_mode == RTE_IOVA_DC) {
> 
>> 
>> 
>> >> +               rte_errno = EINVAL;
>> >> +               return -1;
>> >> +       }
>> >> +
>> >>         if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) {
>> >>                 rte_eal_init_alert("Cannot use IOVA as 'PA' since
>> >> physical addresses are not available");
>> >>                 rte_errno = EINVAL;
>> >> --
>> >> 2.25.0
>> >>

Re: [dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option

[dwilder]

On 2020-03-23 23:19, Jerin Jacob wrote:
> On Mon, Mar 23, 2020 at 11:11 PM dwilder <dwilder@us.ibm.com> wrote:
>> 
>> Thanks you for your review Jerin.  See my responses are inline.
>> 
>> On 2020-03-20 06:24, Jerin Jacob wrote:
>> > On Fri, Feb 21, 2020 at 4:22 AM David Wilder <dwilder@us.ibm.com>
>> > wrote:
>> >>
>> >> If --no-huge is set and iova-mode has not been specified force VA
>> >> mode.
>> >> If --no-huge and --iova-mode=PA is requested error out as this is
>> >> an impossible configuration.
>> >>
>> >> Signed-off-by: David Wilder <dwilder@us.ibm.com>
>> >> ---
>> >>  lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++
>> >>  1 file changed, 14 insertions(+)
>> >>
>> >> diff --git a/lib/librte_eal/linux/eal/eal.c
>> >> b/lib/librte_eal/linux/eal/eal.c
>> >> index 9530ee55f..d3a0a1731 100644
>> >> --- a/lib/librte_eal/linux/eal/eal.c
>> >> +++ b/lib/librte_eal/linux/eal/eal.c
>> >> @@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv)
>> >>
>> >>         /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme
>> >> */
>> >>         if (internal_config.iova_mode == RTE_IOVA_DC) {
>> >> +
>> >>                 /* autodetect the IOVA mapping mode */
>> >>                 enum rte_iova_mode iova_mode =
>> >> rte_bus_get_iommu_class();
>> >>
>> >> +               if (iova_mode == RTE_IOVA_PA &&
>> >> !rte_eal_has_hugepages()) {
>> >> +                       iova_mode = RTE_IOVA_VA;
>> 
>> >
>> > What if igb_uio or vfio_nommu has been loaded(i.e no iommu support
>> > enabled from the driver)? This would fail.
>> 
>> Yes they would fail.  If igb_uio or vfio_nommu (or any driver) cant be
>> forced to VA mode it cant be used with out hugepages.  Drivers can be
>> available but not used therefor we print a warning message.
> 
> I think, the warning will not be enough as the system will fail anyway.
> 
> iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == 
> 1
> case, we need to return error.
> 
> iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == 
> 0
> case warning is enough.
> 

I have a simpler solution.

The goal here is to make --no-huge work when at least one bus/driver 
wants PA mode.
A user can always override the selected mode with --iova-mode=va.
So why not just make --no-huge the same as "--no-huge --iovs-mode=va" ?

I am thinking:

@@ -1060,6 +1060,11 @@ rte_eal_init(int argc, char **argv)

         phys_addrs = rte_eal_using_phys_addrs() != 0;

+       if (!phys_addrs) {
+               internal_config.iova_mode = RTE_IOVA_VA;
+               RTE_LOG(INFO, EAL, "Physical addresses are unavailable, 
selecting IOVA as VA mode.\n");
+       }
+
         /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme 
*/
         if (internal_config.iova_mode == RTE_IOVA_DC) {
                 /* autodetect the IOVA mapping mode */

If a device cant run in VA mode, it will fail to initialize and report 
why.

>> 
>> >
>> >> +                       RTE_LOG(WARNING, EAL, "Some buses want 'PA'
>> >> but forcing 'VA' because --no-huge is requested.\n");
>> >> +                       RTE_LOG(WARNING, EAL, "Not all buses may be
>> >> able to initialize.\n");
>> >> +               }
>> >> +
>> >>                 if (iova_mode == RTE_IOVA_DC) {
>> >>                         RTE_LOG(DEBUG, EAL, "Buses did not request a
>> >> specific IOVA mode.\n");
>> >>
>> >> @@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv)
>> >>                         internal_config.iova_mode;
>> >>         }
>> >>
>> >> +       if (rte_eal_iova_mode() == RTE_IOVA_PA &&
>> >> +           rte_eal_has_hugepages() == 0) {
>> >> +               rte_eal_init_alert("Cannot use IOVA as 'PA' with
>> >> --no-huge");
>> >
>> > Top of the tree already detecting this case. am I missing anything?
>> >
>> > [master]dell[dpdk.org] $ sudo ./build/app/test/dpdk-test  -c 0x3
>> > --no-huge --iova-mode=pa
>> > EAL: Detected 56 lcore(s)
>> > EAL: Detected 2 NUMA nodes
>> > EAL: Static memory layout is selected, amount of reserved memory can
>> > be adjusted with -m or --socket-mem
>> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> > EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not
>> > available
>> > EAL: Cannot use IOVA as 'PA' since physical addresses are not available
>> >
>> 
>> The check you reference is reporting that physical address are not
>> available, for example no permissions to read /proc/self/pagemap.  In
>> this case, if --no-huge is set then PA mode is not allowed. There is 
>> no
>> guarantee that physical address are persistent with out using 
>> hugepages.
> 
> Since this check is under the following, Yes, make sense for the check.
> The old command has explicit  --iova-mode=pa. So it is in the
> different code paths.
> 
> /* if no EAL option "--iova-mode=<pa|va>", use bus IOVA scheme */
>         if (internal_config.iova_mode == RTE_IOVA_DC) {
> 
>> 
>> 
>> >> +               rte_errno = EINVAL;
>> >> +               return -1;
>> >> +       }
>> >> +
>> >>         if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) {
>> >>                 rte_eal_init_alert("Cannot use IOVA as 'PA' since
>> >> physical addresses are not available");
>> >>                 rte_errno = EINVAL;
>> >> --
>> >> 2.25.0
>> >>