drops while transmitting to the kni using rte_kni_tx_burst()

drops while transmitting to the kni using rte_kni_tx_burst()

[Shirley Avishour]

Hi,
I have an application over dpdk which is consisted of the following threads
each running on a separate core:
1) rx thread which listens on in a poll mode for traffic
2) 2 packet processing threads (for load balancing)
3) kni thread (which also runs on a separate core).

the rx thread receives packets and clones them and transmit a copy to the
kni and the other packet is sent to the packet processing unit (hashing
over 2 threads).
the receive traffic rate is 100Mbps.
When working with single packet processing thread I am able to get all the
100Mbps towards the kni with no drops.
but when I activate my application with 2 packet processing threads I start
facing drops towards the kni.
the way I see it the only difference now is that I have another threads
which handles an mbuf and frees it once processing is completed.
Can anyone assist with this case please?

Thanks!

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Ferruh Yigit]

On 1/16/2017 12:20 PM, Shirley Avishour wrote:
> Hi,
> I have an application over dpdk which is consisted of the following threads
> each running on a separate core:
> 1) rx thread which listens on in a poll mode for traffic
> 2) 2 packet processing threads (for load balancing)
> 3) kni thread (which also runs on a separate core).

This is kernel thread, right? Is it bind to any specific core?
Is it possible that this thread shares the core with 2nd processing
thread when enabled?

> 
> the rx thread receives packets and clones them and transmit a copy to the
> kni and the other packet is sent to the packet processing unit (hashing
> over 2 threads).
> the receive traffic rate is 100Mbps.
> When working with single packet processing thread I am able to get all the
> 100Mbps towards the kni with no drops.
> but when I activate my application with 2 packet processing threads I start
> facing drops towards the kni.
> the way I see it the only difference now is that I have another threads
> which handles an mbuf and frees it once processing is completed.
> Can anyone assist with this case please?
> 
> Thanks!
> 

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Shirley Avishour]

Hi,
As I wrote the kernel thread runs on a dedicated lcore.
running top while my application is running I see kni_single and the cpu
usage is really low...
Is there any rate limitation for transmitting to the kernel interface
(since packets are being copied in the kernel).


On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 1/16/2017 12:20 PM, Shirley Avishour wrote:
> > Hi,
> > I have an application over dpdk which is consisted of the following
> threads
> > each running on a separate core:
> > 1) rx thread which listens on in a poll mode for traffic
> > 2) 2 packet processing threads (for load balancing)
> > 3) kni thread (which also runs on a separate core).
>
> This is kernel thread, right? Is it bind to any specific core?
> Is it possible that this thread shares the core with 2nd processing
> thread when enabled?
>
> >
> > the rx thread receives packets and clones them and transmit a copy to the
> > kni and the other packet is sent to the packet processing unit (hashing
> > over 2 threads).
> > the receive traffic rate is 100Mbps.
> > When working with single packet processing thread I am able to get all
> the
> > 100Mbps towards the kni with no drops.
> > but when I activate my application with 2 packet processing threads I
> start
> > facing drops towards the kni.
> > the way I see it the only difference now is that I have another threads
> > which handles an mbuf and frees it once processing is completed.
> > Can anyone assist with this case please?
> >
> > Thanks!
> >
>
>

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Ferruh Yigit]

On 1/16/2017 2:47 PM, Shirley Avishour wrote:
> Hi,
> As I wrote the kernel thread runs on a dedicated lcore.
> running top while my application is running I see kni_single and the cpu
> usage is really low...
> Is there any rate limitation for transmitting to the kernel interface
> (since packets are being copied in the kernel).

Yes, kind of, kernel thread sleeps periodically, with a value defined by
KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this value, if you
want thread do more work, less sleep J

Also KNI_RX_LOOP_NUM can be updated for same purpose.

> 
> 
> On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>> wrote:
> 
>     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
>     > Hi,
>     > I have an application over dpdk which is consisted of the following threads
>     > each running on a separate core:
>     > 1) rx thread which listens on in a poll mode for traffic
>     > 2) 2 packet processing threads (for load balancing)
>     > 3) kni thread (which also runs on a separate core).
> 
>     This is kernel thread, right? Is it bind to any specific core?
>     Is it possible that this thread shares the core with 2nd processing
>     thread when enabled?
> 
>     >
>     > the rx thread receives packets and clones them and transmit a copy
>     to the
>     > kni and the other packet is sent to the packet processing unit
>     (hashing
>     > over 2 threads).
>     > the receive traffic rate is 100Mbps.
>     > When working with single packet processing thread I am able to get
>     all the
>     > 100Mbps towards the kni with no drops.
>     > but when I activate my application with 2 packet processing
>     threads I start
>     > facing drops towards the kni.
>     > the way I see it the only difference now is that I have another
>     threads
>     > which handles an mbuf and frees it once processing is completed.
>     > Can anyone assist with this case please?
>     >
>     > Thanks!
>     >
> 
> 

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Shirley Avishour]

I am currently using the kernel interface for recording the received
traffic by duplicating the received packets and sending a copy to the kni
(and performing pcap_open_live on the kni).
my goal rate is around 500Mbps. is it possible to achieve it via the kni ??


On Mon, Jan 16, 2017 at 4:55 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 1/16/2017 2:47 PM, Shirley Avishour wrote:
> > Hi,
> > As I wrote the kernel thread runs on a dedicated lcore.
> > running top while my application is running I see kni_single and the cpu
> > usage is really low...
> > Is there any rate limitation for transmitting to the kernel interface
> > (since packets are being copied in the kernel).
>
> Yes, kind of, kernel thread sleeps periodically, with a value defined by
> KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this value, if you
> want thread do more work, less sleep J
>
> Also KNI_RX_LOOP_NUM can be updated for same purpose.
>
> >
> >
> > On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com
> > <mailto:ferruh.yigit@intel.com>> wrote:
> >
> >     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
> >     > Hi,
> >     > I have an application over dpdk which is consisted of the
> following threads
> >     > each running on a separate core:
> >     > 1) rx thread which listens on in a poll mode for traffic
> >     > 2) 2 packet processing threads (for load balancing)
> >     > 3) kni thread (which also runs on a separate core).
> >
> >     This is kernel thread, right? Is it bind to any specific core?
> >     Is it possible that this thread shares the core with 2nd processing
> >     thread when enabled?
> >
> >     >
> >     > the rx thread receives packets and clones them and transmit a copy
> >     to the
> >     > kni and the other packet is sent to the packet processing unit
> >     (hashing
> >     > over 2 threads).
> >     > the receive traffic rate is 100Mbps.
> >     > When working with single packet processing thread I am able to get
> >     all the
> >     > 100Mbps towards the kni with no drops.
> >     > but when I activate my application with 2 packet processing
> >     threads I start
> >     > facing drops towards the kni.
> >     > the way I see it the only difference now is that I have another
> >     threads
> >     > which handles an mbuf and frees it once processing is completed.
> >     > Can anyone assist with this case please?
> >     >
> >     > Thanks!
> >     >
> >
> >
>
>

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Shirley Avishour]

Hi,
KNI_KTHREAD_RESCHEDULE_INTERVAL is currently set to 5usec. how should I
tweak this value to get better performance?
and can you explain the use of KNI_RX_LOOP_NUM and what can I possibly
modify it to ?

Thanks!


On Mon, Jan 16, 2017 at 4:55 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 1/16/2017 2:47 PM, Shirley Avishour wrote:
> > Hi,
> > As I wrote the kernel thread runs on a dedicated lcore.
> > running top while my application is running I see kni_single and the cpu
> > usage is really low...
> > Is there any rate limitation for transmitting to the kernel interface
> > (since packets are being copied in the kernel).
>
> Yes, kind of, kernel thread sleeps periodically, with a value defined by
> KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this value, if you
> want thread do more work, less sleep J
>
> Also KNI_RX_LOOP_NUM can be updated for same purpose.
>
> >
> >
> > On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com
> > <mailto:ferruh.yigit@intel.com>> wrote:
> >
> >     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
> >     > Hi,
> >     > I have an application over dpdk which is consisted of the
> following threads
> >     > each running on a separate core:
> >     > 1) rx thread which listens on in a poll mode for traffic
> >     > 2) 2 packet processing threads (for load balancing)
> >     > 3) kni thread (which also runs on a separate core).
> >
> >     This is kernel thread, right? Is it bind to any specific core?
> >     Is it possible that this thread shares the core with 2nd processing
> >     thread when enabled?
> >
> >     >
> >     > the rx thread receives packets and clones them and transmit a copy
> >     to the
> >     > kni and the other packet is sent to the packet processing unit
> >     (hashing
> >     > over 2 threads).
> >     > the receive traffic rate is 100Mbps.
> >     > When working with single packet processing thread I am able to get
> >     all the
> >     > 100Mbps towards the kni with no drops.
> >     > but when I activate my application with 2 packet processing
> >     threads I start
> >     > facing drops towards the kni.
> >     > the way I see it the only difference now is that I have another
> >     threads
> >     > which handles an mbuf and frees it once processing is completed.
> >     > Can anyone assist with this case please?
> >     >
> >     > Thanks!
> >     >
> >
> >
>
>

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Shirley Avishour]

Hi,
can the KNI_KTHREAD_RESCHEDULE_INTERVAL decrease to lower values than 5
usecs?? is it effective at all?
and what is the purpose of KNI_RX_LOOP_NUM? I am not sure I understand what
it does and what should I do with it to improve the performance.

thanks!

On Mon, Jan 16, 2017 at 5:43 PM, Shirley Avishour <shirley@imvisiontech.com>
wrote:

> Hi,
> KNI_KTHREAD_RESCHEDULE_INTERVAL is currently set to 5usec. how should I
> tweak this value to get better performance?
> and can you explain the use of KNI_RX_LOOP_NUM and what can I possibly
> modify it to ?
>
> Thanks!
>
>
> On Mon, Jan 16, 2017 at 4:55 PM, Ferruh Yigit <ferruh.yigit@intel.com>
> wrote:
>
>> On 1/16/2017 2:47 PM, Shirley Avishour wrote:
>> > Hi,
>> > As I wrote the kernel thread runs on a dedicated lcore.
>> > running top while my application is running I see kni_single and the cpu
>> > usage is really low...
>> > Is there any rate limitation for transmitting to the kernel interface
>> > (since packets are being copied in the kernel).
>>
>> Yes, kind of, kernel thread sleeps periodically, with a value defined by
>> KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this value, if you
>> want thread do more work, less sleep J
>>
>> Also KNI_RX_LOOP_NUM can be updated for same purpose.
>>
>> >
>> >
>> > On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com
>> > <mailto:ferruh.yigit@intel.com>> wrote:
>> >
>> >     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
>> >     > Hi,
>> >     > I have an application over dpdk which is consisted of the
>> following threads
>> >     > each running on a separate core:
>> >     > 1) rx thread which listens on in a poll mode for traffic
>> >     > 2) 2 packet processing threads (for load balancing)
>> >     > 3) kni thread (which also runs on a separate core).
>> >
>> >     This is kernel thread, right? Is it bind to any specific core?
>> >     Is it possible that this thread shares the core with 2nd processing
>> >     thread when enabled?
>> >
>> >     >
>> >     > the rx thread receives packets and clones them and transmit a copy
>> >     to the
>> >     > kni and the other packet is sent to the packet processing unit
>> >     (hashing
>> >     > over 2 threads).
>> >     > the receive traffic rate is 100Mbps.
>> >     > When working with single packet processing thread I am able to get
>> >     all the
>> >     > 100Mbps towards the kni with no drops.
>> >     > but when I activate my application with 2 packet processing
>> >     threads I start
>> >     > facing drops towards the kni.
>> >     > the way I see it the only difference now is that I have another
>> >     threads
>> >     > which handles an mbuf and frees it once processing is completed.
>> >     > Can anyone assist with this case please?
>> >     >
>> >     > Thanks!
>> >     >
>> >
>> >
>>
>>
>

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Jay Rolette]

On Mon, Jan 16, 2017 at 8:55 AM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 1/16/2017 2:47 PM, Shirley Avishour wrote:
> > Hi,
> > As I wrote the kernel thread runs on a dedicated lcore.
> > running top while my application is running I see kni_single and the cpu
> > usage is really low...
> > Is there any rate limitation for transmitting to the kernel interface
> > (since packets are being copied in the kernel).
>
> Yes, kind of, kernel thread sleeps periodically, with a value defined by
> KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this value, if you
> want thread do more work, less sleep J
>

YMMV, but that is unlikely to be the problem. See this thread for why the
code is misleading if you don't understand what the schedule_timeout() call
is doing:

http://dpdk.org/ml/archives/dev/2015-June/018858.html

Jay


>
> Also KNI_RX_LOOP_NUM can be updated for same purpose.
>
> >
> >
> > On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com
> > <mailto:ferruh.yigit@intel.com>> wrote:
> >
> >     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
> >     > Hi,
> >     > I have an application over dpdk which is consisted of the
> following threads
> >     > each running on a separate core:
> >     > 1) rx thread which listens on in a poll mode for traffic
> >     > 2) 2 packet processing threads (for load balancing)
> >     > 3) kni thread (which also runs on a separate core).
> >
> >     This is kernel thread, right? Is it bind to any specific core?
> >     Is it possible that this thread shares the core with 2nd processing
> >     thread when enabled?
> >
> >     >
> >     > the rx thread receives packets and clones them and transmit a copy
> >     to the
> >     > kni and the other packet is sent to the packet processing unit
> >     (hashing
> >     > over 2 threads).
> >     > the receive traffic rate is 100Mbps.
> >     > When working with single packet processing thread I am able to get
> >     all the
> >     > 100Mbps towards the kni with no drops.
> >     > but when I activate my application with 2 packet processing
> >     threads I start
> >     > facing drops towards the kni.
> >     > the way I see it the only difference now is that I have another
> >     threads
> >     > which handles an mbuf and frees it once processing is completed.
> >     > Can anyone assist with this case please?
> >     >
> >     > Thanks!
> >     >
> >
> >
>
>

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Ferruh Yigit]

On 1/16/2017 2:58 PM, Shirley Avishour wrote:
> I am currently using the kernel interface for recording the received
> traffic by duplicating the received packets and sending a copy to the
> kni (and performing pcap_open_live on the kni).
> my goal rate is around 500Mbps. is it possible to achieve it via the kni ??

According quick experiment,
- with kni module lo_mode_fifo_skb (which send all received packets to
tx, but allocating and copying data to skb to be more realistic)
- single kernel thread
- kernel thread bind to a core
- using kni sample app
- With small packets

Best numbers get when rx,tx and kernel cores are in same socket with the
NIC, it is ~1.7Mpps (million packet per second)

When KNI_RX_LOOP_NUM increased to 10000, it becomes ~1.9Mpps.

> 
> 
> On Mon, Jan 16, 2017 at 4:55 PM, Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>> wrote:
> 
>     On 1/16/2017 2:47 PM, Shirley Avishour wrote:
>     > Hi,
>     > As I wrote the kernel thread runs on a dedicated lcore.
>     > running top while my application is running I see kni_single and the cpu
>     > usage is really low...
>     > Is there any rate limitation for transmitting to the kernel interface
>     > (since packets are being copied in the kernel).
> 
>     Yes, kind of, kernel thread sleeps periodically, with a value defined by
>     KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this value, if you
>     want thread do more work, less sleep J
> 
>     Also KNI_RX_LOOP_NUM can be updated for same purpose.
> 
>     >
>     >
>     > On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>
>     > <mailto:ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>>>
>     wrote:
>     >
>     >     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
>     >     > Hi,
>     >     > I have an application over dpdk which is consisted of the
>     following threads
>     >     > each running on a separate core:
>     >     > 1) rx thread which listens on in a poll mode for traffic
>     >     > 2) 2 packet processing threads (for load balancing)
>     >     > 3) kni thread (which also runs on a separate core).
>     >
>     >     This is kernel thread, right? Is it bind to any specific core?
>     >     Is it possible that this thread shares the core with 2nd
>     processing
>     >     thread when enabled?
>     >
>     >     >
>     >     > the rx thread receives packets and clones them and transmit
>     a copy
>     >     to the
>     >     > kni and the other packet is sent to the packet processing unit
>     >     (hashing
>     >     > over 2 threads).
>     >     > the receive traffic rate is 100Mbps.
>     >     > When working with single packet processing thread I am able
>     to get
>     >     all the
>     >     > 100Mbps towards the kni with no drops.
>     >     > but when I activate my application with 2 packet processing
>     >     threads I start
>     >     > facing drops towards the kni.
>     >     > the way I see it the only difference now is that I have another
>     >     threads
>     >     > which handles an mbuf and frees it once processing is completed.
>     >     > Can anyone assist with this case please?
>     >     >
>     >     > Thanks!
>     >     >
>     >
>     >
> 
> 

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Ferruh Yigit]

On 1/17/2017 12:34 PM, Shirley Avishour wrote:
> Hi,
> can the KNI_KTHREAD_RESCHEDULE_INTERVAL decrease to lower values than 5
> usecs?? is it effective at all?

You are right, it may not be since metric is usecs, and
usecs_to_jiffies() most probably will give same value for 5 and lower.

Removing that sleep completely (CONFIG_RTE_KNI_PREEMPT_DEFAULT=n) helps
a little.

> and what is the purpose of KNI_RX_LOOP_NUM? I am not sure I understand
> what it does and what should I do with it to improve the performance.

Increasing the loop number, also increases the work/sleep ratio, and
increasing the performance a little.

> 
> thanks!
> 
> On Mon, Jan 16, 2017 at 5:43 PM, Shirley Avishour
> <shirley@imvisiontech.com <mailto:shirley@imvisiontech.com>> wrote:
> 
>     Hi,
>     KNI_KTHREAD_RESCHEDULE_INTERVAL is currently set to 5usec. how
>     should I tweak this value to get better performance?
>     and can you explain the use of KNI_RX_LOOP_NUM and what can I
>     possibly modify it to ?
> 
>     Thanks!
> 
> 
>     On Mon, Jan 16, 2017 at 4:55 PM, Ferruh Yigit
>     <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>> wrote:
> 
>         On 1/16/2017 2:47 PM, Shirley Avishour wrote:
>         > Hi,
>         > As I wrote the kernel thread runs on a dedicated lcore.
>         > running top while my application is running I see kni_single and the cpu
>         > usage is really low...
>         > Is there any rate limitation for transmitting to the kernel interface
>         > (since packets are being copied in the kernel).
> 
>         Yes, kind of, kernel thread sleeps periodically, with a value
>         defined by
>         KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this
>         value, if you
>         want thread do more work, less sleep J
> 
>         Also KNI_RX_LOOP_NUM can be updated for same purpose.
> 
>         >
>         >
>         > On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>
>         > <mailto:ferruh.yigit@intel.com
>         <mailto:ferruh.yigit@intel.com>>> wrote:
>         >
>         >     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
>         >     > Hi,
>         >     > I have an application over dpdk which is consisted of
>         the following threads
>         >     > each running on a separate core:
>         >     > 1) rx thread which listens on in a poll mode for traffic
>         >     > 2) 2 packet processing threads (for load balancing)
>         >     > 3) kni thread (which also runs on a separate core).
>         >
>         >     This is kernel thread, right? Is it bind to any specific core?
>         >     Is it possible that this thread shares the core with 2nd
>         processing
>         >     thread when enabled?
>         >
>         >     >
>         >     > the rx thread receives packets and clones them and
>         transmit a copy
>         >     to the
>         >     > kni and the other packet is sent to the packet
>         processing unit
>         >     (hashing
>         >     > over 2 threads).
>         >     > the receive traffic rate is 100Mbps.
>         >     > When working with single packet processing thread I am
>         able to get
>         >     all the
>         >     > 100Mbps towards the kni with no drops.
>         >     > but when I activate my application with 2 packet processing
>         >     threads I start
>         >     > facing drops towards the kni.
>         >     > the way I see it the only difference now is that I have
>         another
>         >     threads
>         >     > which handles an mbuf and frees it once processing is
>         completed.
>         >     > Can anyone assist with this case please?
>         >     >
>         >     > Thanks!
>         >     >
>         >
>         >
> 
> 
> 

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Ferruh Yigit]

On 1/17/2017 5:46 PM, Ferruh Yigit wrote:
> On 1/16/2017 2:58 PM, Shirley Avishour wrote:
>> I am currently using the kernel interface for recording the received
>> traffic by duplicating the received packets and sending a copy to the
>> kni (and performing pcap_open_live on the kni).
>> my goal rate is around 500Mbps. is it possible to achieve it via the kni ??
> 
> According quick experiment,
> - with kni module lo_mode_fifo_skb (which send all received packets to
> tx, but allocating and copying data to skb to be more realistic)
> - single kernel thread
> - kernel thread bind to a core
> - using kni sample app
> - With small packets
> 
> Best numbers get when rx,tx and kernel cores are in same socket with the
> NIC, it is ~1.7Mpps (million packet per second)
> 
> When KNI_RX_LOOP_NUM increased to 10000, it becomes ~1.9Mpps.
> 

And again, very quick test, between two KNI ports, with kni sample app,
using iperf default values, gives ~3 Gbits/sec

>>
>>
>> On Mon, Jan 16, 2017 at 4:55 PM, Ferruh Yigit <ferruh.yigit@intel.com
>> <mailto:ferruh.yigit@intel.com>> wrote:
>>
>>     On 1/16/2017 2:47 PM, Shirley Avishour wrote:
>>     > Hi,
>>     > As I wrote the kernel thread runs on a dedicated lcore.
>>     > running top while my application is running I see kni_single and the cpu
>>     > usage is really low...
>>     > Is there any rate limitation for transmitting to the kernel interface
>>     > (since packets are being copied in the kernel).
>>
>>     Yes, kind of, kernel thread sleeps periodically, with a value defined by
>>     KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this value, if you
>>     want thread do more work, less sleep J
>>
>>     Also KNI_RX_LOOP_NUM can be updated for same purpose.
>>
>>     >
>>     >
>>     > On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>
>>     > <mailto:ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>>>
>>     wrote:
>>     >
>>     >     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
>>     >     > Hi,
>>     >     > I have an application over dpdk which is consisted of the
>>     following threads
>>     >     > each running on a separate core:
>>     >     > 1) rx thread which listens on in a poll mode for traffic
>>     >     > 2) 2 packet processing threads (for load balancing)
>>     >     > 3) kni thread (which also runs on a separate core).
>>     >
>>     >     This is kernel thread, right? Is it bind to any specific core?
>>     >     Is it possible that this thread shares the core with 2nd
>>     processing
>>     >     thread when enabled?
>>     >
>>     >     >
>>     >     > the rx thread receives packets and clones them and transmit
>>     a copy
>>     >     to the
>>     >     > kni and the other packet is sent to the packet processing unit
>>     >     (hashing
>>     >     > over 2 threads).
>>     >     > the receive traffic rate is 100Mbps.
>>     >     > When working with single packet processing thread I am able
>>     to get
>>     >     all the
>>     >     > 100Mbps towards the kni with no drops.
>>     >     > but when I activate my application with 2 packet processing
>>     >     threads I start
>>     >     > facing drops towards the kni.
>>     >     > the way I see it the only difference now is that I have another
>>     >     threads
>>     >     > which handles an mbuf and frees it once processing is completed.
>>     >     > Can anyone assist with this case please?
>>     >     >
>>     >     > Thanks!
>>     >     >
>>     >
>>     >
>>
>>
> 

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Shirley Avishour]

Hi,
thanks for the quick answer.
I ran the same test with one kni port
with the following configuration:

sudo insmod rte_kni.ko lo_mode=lo_mode_fifo_skb

sudo ./kni -c 0xf -n 4 -- -P p 0x1 --config="(0,1,2,3)"

and transmitted towards the single port traffic with rate of 300Mbps.
as you can see below I have packet drops:
what is the difference between what you tested and what I see below?
is there some other configuration or adjustments to be done on the kni that
I am missing?


======  ==============  ============  ============  ============
============

**KNI example application statistics**
======  ==============  ============  ============  ============
============
 Port    Lcore(RX/TX)    rx_packets    rx_dropped    tx_packets
tx_dropped
------  --------------  ------------  ------------  ------------
------------
      0          1/ 2     106059534         21978     106058868
0
======  ==============  ============  ============  ============
============

**KNI example application statistics**
======  ==============  ============  ============  ============
============
 Port    Lcore(RX/TX)    rx_packets    rx_dropped    tx_packets
tx_dropped
------  --------------  ------------  ------------  ------------
------------
      0          1/ 2     106137710         21995     106137054
0
======  ==============  ============  ============  ============
============

**KNI example application statistics**
======  ==============  ============  ============  ============
============
 Port    Lcore(RX/TX)    rx_packets    rx_dropped    tx_packets
tx_dropped
------  --------------  ------------  ------------  ------------
------------
      0          1/ 2     106217479         21995     106216603
0
======  ==============  ============  ============  ============
============

**KNI example application statistics**
======  ==============  ============  ============  ============
============
 Port    Lcore(RX/TX)    rx_packets    rx_dropped    tx_packets
tx_dropped
------  --------------  ------------  ------------  ------------
------------
      0          1/ 2     116349769         24150     116349796
0
======  ==============  ============  ============  ============
============

**KNI example application statistics**
======  ==============  ============  ============  ============
============
 Port    Lcore(RX/TX)    rx_packets    rx_dropped    tx_packets
tx_dropped
------  --------------  ------------  ------------  ------------
------------
      0          1/ 2     116349769         24150     116349796
0
======  ==============  ============  ============  ============
============
^[[A
**KNI example application statistics**
======  ==============  ============  ============  ============
============
 Port    Lcore(RX/TX)    rx_packets    rx_dropped    tx_packets
tx_dropped
------  --------------  ------------  ------------  ------------
------------
      0          1/ 2     117954991         24530     117954361
0
======  ==============  ============  ============  ============
============


On Tue, Jan 17, 2017 at 7:57 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 1/17/2017 5:46 PM, Ferruh Yigit wrote:
> > On 1/16/2017 2:58 PM, Shirley Avishour wrote:
> >> I am currently using the kernel interface for recording the received
> >> traffic by duplicating the received packets and sending a copy to the
> >> kni (and performing pcap_open_live on the kni).
> >> my goal rate is around 500Mbps. is it possible to achieve it via the
> kni ??
> >
> > According quick experiment,
> > - with kni module lo_mode_fifo_skb (which send all received packets to
> > tx, but allocating and copying data to skb to be more realistic)
> > - single kernel thread
> > - kernel thread bind to a core
> > - using kni sample app
> > - With small packets
> >
> > Best numbers get when rx,tx and kernel cores are in same socket with the
> > NIC, it is ~1.7Mpps (million packet per second)
> >
> > When KNI_RX_LOOP_NUM increased to 10000, it becomes ~1.9Mpps.
> >
>
> And again, very quick test, between two KNI ports, with kni sample app,
> using iperf default values, gives ~3 Gbits/sec
>
> >>
> >>
> >> On Mon, Jan 16, 2017 at 4:55 PM, Ferruh Yigit <ferruh.yigit@intel.com
> >> <mailto:ferruh.yigit@intel.com>> wrote:
> >>
> >>     On 1/16/2017 2:47 PM, Shirley Avishour wrote:
> >>     > Hi,
> >>     > As I wrote the kernel thread runs on a dedicated lcore.
> >>     > running top while my application is running I see kni_single and
> the cpu
> >>     > usage is really low...
> >>     > Is there any rate limitation for transmitting to the kernel
> interface
> >>     > (since packets are being copied in the kernel).
> >>
> >>     Yes, kind of, kernel thread sleeps periodically, with a value
> defined by
> >>     KNI_KTHREAD_RESCHEDULE_INTERVAL. You can try tweaking this value,
> if you
> >>     want thread do more work, less sleep J
> >>
> >>     Also KNI_RX_LOOP_NUM can be updated for same purpose.
> >>
> >>     >
> >>     >
> >>     > On Mon, Jan 16, 2017 at 4:42 PM, Ferruh Yigit <
> ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>
> >>     > <mailto:ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>>>
> >>     wrote:
> >>     >
> >>     >     On 1/16/2017 12:20 PM, Shirley Avishour wrote:
> >>     >     > Hi,
> >>     >     > I have an application over dpdk which is consisted of the
> >>     following threads
> >>     >     > each running on a separate core:
> >>     >     > 1) rx thread which listens on in a poll mode for traffic
> >>     >     > 2) 2 packet processing threads (for load balancing)
> >>     >     > 3) kni thread (which also runs on a separate core).
> >>     >
> >>     >     This is kernel thread, right? Is it bind to any specific core?
> >>     >     Is it possible that this thread shares the core with 2nd
> >>     processing
> >>     >     thread when enabled?
> >>     >
> >>     >     >
> >>     >     > the rx thread receives packets and clones them and transmit
> >>     a copy
> >>     >     to the
> >>     >     > kni and the other packet is sent to the packet processing
> unit
> >>     >     (hashing
> >>     >     > over 2 threads).
> >>     >     > the receive traffic rate is 100Mbps.
> >>     >     > When working with single packet processing thread I am able
> >>     to get
> >>     >     all the
> >>     >     > 100Mbps towards the kni with no drops.
> >>     >     > but when I activate my application with 2 packet processing
> >>     >     threads I start
> >>     >     > facing drops towards the kni.
> >>     >     > the way I see it the only difference now is that I have
> another
> >>     >     threads
> >>     >     > which handles an mbuf and frees it once processing is
> completed.
> >>     >     > Can anyone assist with this case please?
> >>     >     >
> >>     >     > Thanks!
> >>     >     >
> >>     >
> >>     >
> >>
> >>
> >
>
>

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Jason Kwon]

Sorry to hijack the thread, but I was wondering, is the binding of the KNI
kernel thread only applicable in multiple kthread mode?  Does it apply in
single kernel thread mode?  Looking at the 16.07.2 source, I saw the
following:

        /**
         * Create a new kernel thread for multiple mode, set its core
affinity,
         * and finally wake it up.
         */
        if (multiple_kthread_on) {
                kni->pthread = kthread_create(kni_thread_multiple,
                                              (void *)kni,
                                              "kni_%s", kni->name);
                if (IS_ERR(kni->pthread)) {
                        kni_dev_remove(kni);
                        return -ECANCELED;
                }
                if (dev_info.force_bind)
                        kthread_bind(kni->pthread, kni->core_id);
                wake_up_process(kni->pthread);
        }

Thanks,

Jason

On Mon, Jan 16, 2017 at 6:42 AM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 1/16/2017 12:20 PM, Shirley Avishour wrote:
> > Hi,
> > I have an application over dpdk which is consisted of the following
> threads
> > each running on a separate core:
> > 1) rx thread which listens on in a poll mode for traffic
> > 2) 2 packet processing threads (for load balancing)
> > 3) kni thread (which also runs on a separate core).
>
> This is kernel thread, right? Is it bind to any specific core?
> Is it possible that this thread shares the core with 2nd processing
> thread when enabled?
>
> >
> > the rx thread receives packets and clones them and transmit a copy to the
> > kni and the other packet is sent to the packet processing unit (hashing
> > over 2 threads).
> > the receive traffic rate is 100Mbps.
> > When working with single packet processing thread I am able to get all
> the
> > 100Mbps towards the kni with no drops.
> > but when I activate my application with 2 packet processing threads I
> start
> > facing drops towards the kni.
> > the way I see it the only difference now is that I have another threads
> > which handles an mbuf and frees it once processing is completed.
> > Can anyone assist with this case please?
> >
> > Thanks!
> >
>
>

Re: drops while transmitting to the kni using rte_kni_tx_burst()

[Shirley Avishour]

Hi Jason,
You are right but I've tried to work in multiple mode as well and it did
not change the result.
I hope this issue will be solved.

Thanks,
Shirley.

On Fri, Jan 20, 2017 at 9:48 PM, Jason Kwon <m43kwon@gmail.com> wrote:

> Sorry to hijack the thread, but I was wondering, is the binding of the KNI
> kernel thread only applicable in multiple kthread mode?  Does it apply in
> single kernel thread mode?  Looking at the 16.07.2 source, I saw the
> following:
>
>         /**
>          * Create a new kernel thread for multiple mode, set its core
> affinity,
>          * and finally wake it up.
>          */
>         if (multiple_kthread_on) {
>                 kni->pthread = kthread_create(kni_thread_multiple,
>                                               (void *)kni,
>                                               "kni_%s", kni->name);
>                 if (IS_ERR(kni->pthread)) {
>                         kni_dev_remove(kni);
>                         return -ECANCELED;
>                 }
>                 if (dev_info.force_bind)
>                         kthread_bind(kni->pthread, kni->core_id);
>                 wake_up_process(kni->pthread);
>         }
>
> Thanks,
>
> Jason
>
> On Mon, Jan 16, 2017 at 6:42 AM, Ferruh Yigit <ferruh.yigit@intel.com>
> wrote:
>
>> On 1/16/2017 12:20 PM, Shirley Avishour wrote:
>> > Hi,
>> > I have an application over dpdk which is consisted of the following
>> threads
>> > each running on a separate core:
>> > 1) rx thread which listens on in a poll mode for traffic
>> > 2) 2 packet processing threads (for load balancing)
>> > 3) kni thread (which also runs on a separate core).
>>
>> This is kernel thread, right? Is it bind to any specific core?
>> Is it possible that this thread shares the core with 2nd processing
>> thread when enabled?
>>
>> >
>> > the rx thread receives packets and clones them and transmit a copy to
>> the
>> > kni and the other packet is sent to the packet processing unit (hashing
>> > over 2 threads).
>> > the receive traffic rate is 100Mbps.
>> > When working with single packet processing thread I am able to get all
>> the
>> > 100Mbps towards the kni with no drops.
>> > but when I activate my application with 2 packet processing threads I
>> start
>> > facing drops towards the kni.
>> > the way I see it the only difference now is that I have another threads
>> > which handles an mbuf and frees it once processing is completed.
>> > Can anyone assist with this case please?
>> >
>> > Thanks!
>> >
>>
>>
>