RE: [PATCH] testsuite: dpdk: Add l2fwd example which is based on DPDK.

["Chen, Hongzhan" <hongzhan.chen@intel.com>]

>-----Original Message-----
>From: Jan Kiszka <jan.kiszka@siemens.com>
>Sent: Tuesday, January 31, 2023 3:27 PM
>To: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@lists.linux.dev
>Subject: Re: [PATCH] testsuite: dpdk: Add l2fwd example which is based on
>DPDK.
>
>On 31.01.23 06:10, Chen, Hongzhan wrote:
>>
>>
>>> -----Original Message-----
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>> Sent: Thursday, January 19, 2023 1:44 AM
>>> To: Chen, Hongzhan <hongzhan.chen@intel.com>;
>xenomai@lists.linux.dev
>>> Subject: Re: [PATCH] testsuite: dpdk: Add l2fwd example which is based on
>>> DPDK.
>>>
>>> On 18.01.23 14:05, Hongzhan Chen wrote:
>>>> The L2 Forwarding sample application(l2fwd) is a simple
>>>> example of packet processing using the Data Plane
>>>> Development Kit (DPDK).
>>>
>>> Can you elaborate a bit on the envisioned use cases here and go more
>>> into details in the accompanied documentation? The latter only describes
>>> the "how", not at all the "why" (or "which use cases"). Some words on
>>> the threading model (or limitations of it) would surely be good as well
>>> ("how can I integrate this pattern into a real application?").
>>>
>>> Regarding the patch itself, you should go over it again and eliminate
>>> debugging left-overs as well as unrelated changes (there is one at least
>>> in testsuite/Makefile.am). The way of the integration looks fine to me
>>> in principle.
>>>
>>> BTW, what prevents the usage of the VFIO driver so far?
>>
>> I have not looked into and tried VFIO driver so far. Last time you told me
>that we may need to
>> develop or port VFIO related drivers if we want to use VFIO-based dpdk in
>Xenomai but UIO-based
>> does not such issue. I maybe misunderstood what you guided. If the main
>difference between uio and vfio is that vfio is capable of programming the
>platform's IOMMU as described in [1], after vfio init, there is no linux syscall
>involved during receiving and transmitting packets just as UIO case, we can do
>same thing for VFIO with ported l2fwd.
>
>I haven't looked into VFIO myself, but the key question is whether some
>userspace activity in the primary domain could trigger mapping activity
>of the driver in the secondary one. If not, VFIO should be fine. If so,
>and that was the case for some RTnet drivers, we would need to avoid
>that. I was just wondering if you explored the path already and could
>share your experience.

According to my test, the patch works [1] on I255 NIC with vfio-pci module & iommu=pt 
& intel_iommu=on & VT-D enabled in BIOS. There is no mapping 
activities found during receiving and transmitting  packets at least in 
main loop of the l2fwd case after run about 17 hours and  I checked with strace log[2].

Regards

Hongzhan Chen

[1]: cat /proc/xenomai/sched/stat

CPU  PID    MSW        CSW        XSC        PF    STAT       %CPU  NAME
  0  0      0          3          0          0     00218000  100.0  [ROOT/0]
  1  0      0          72641414   0          0     00218000   99.9  [ROOT/1]
  2  0      0          0          0          0     00218000  100.0  [ROOT/2]
  3  0      0          0          0          0     00218000  100.0  [ROOT/3]
  4  0      0          1          0          0     00218000  100.0  [ROOT/4]
  5  0      0          2          0          0     00218000  100.0  [ROOT/5]
  5  935    1          1          5          0     002680c0    0.0  l2fwd
  1  954    6718       13436      6722       0     00268042    0.0  display-935
  1  955    1          67171569   67178287   0     00248044    0.1  l2fwd

[2]: Strace log:

13:36:14.266949 futex(0x7ffed97d59b8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1675258579, tv_nsec=0}, FUTEX_BITSET_MATCH_ANY) = 0
13:36:14.266984 sched_get_priority_min(SCHED_FIFO) = 1
13:36:14.267004 sched_get_priority_max(SCHED_FIFO) = 99
13:36:14.267028 mmap(NULL, 69632, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1e13fef000
13:36:14.267102 mprotect(0x7f1e13ff0000, 65536, PROT_READ|PROT_WRITE) = 0
13:36:14.267347 clone(child_stack=0x7f1e13ffddf0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f1e13fff9d0, tls=0x7f1e13fff700, child_tidptr=0x7f1e13fff9d0) = 955
13:36:14.267380 sched_setaffinity(955, 128, [1]) = 0
13:36:14.267422 futex(0x7ffed97d59b8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1675258579, tv_nsec=0}, FUTEX_BITSET_MATCH_ANY) = -1 EAGAIN (Resource temporarily unavailable)
13:36:14.267480 rt_sigtimedwait([HUP INT ALRM TERM], {si_signo=SIGINT, si_code=SI_KERNEL}, NULL, 8) = 2 (SIGINT)
08:16:50.286528 getpid()                = 935
08:16:50.286556 tgkill(935, 954, SIGRTMIN) = 0
08:16:50.286586 getpid()                = 935

>
>Jan
>
>>
>> Regards
>>
>> Hongzhan Chen
>>
>> [1]:
>https://spdk.io/doc/userspace.html#:~:text=The%20primary%20difference%2
>0between%20uio,User%20Space%20for%20full%20details.
>>>
>>> Jan
>>>
>>> --
>>> Siemens AG, Technology
>>> Competence Center Embedded Linux
>>
>
>--
>Siemens AG, Technology
>Competence Center Embedded Linux
>