* net/i40e: latency issue due fix interrupt throttling setting in PF
@ 2017-11-20 15:18 Hanoch Haim (hhaim)
2017-11-24 11:24 ` Hanoch Haim (hhaim)
0 siblings, 1 reply; 12+ messages in thread
From: Hanoch Haim (hhaim) @ 2017-11-20 15:18 UTC (permalink / raw)
To: dev; +Cc: Wu, Jingjing (jingjing.wu@intel.com), Hanoch Haim (hhaim)
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-20 15:18 net/i40e: latency issue due fix interrupt throttling setting in PF Hanoch Haim (hhaim)
@ 2017-11-24 11:24 ` Hanoch Haim (hhaim)
2017-11-28 12:50 ` Wu, Jingjing
0 siblings, 1 reply; 12+ messages in thread
From: Hanoch Haim (hhaim) @ 2017-11-24 11:24 UTC (permalink / raw)
To: dev; +Cc: Wu, Jingjing (jingjing.wu@intel.com)
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org
Cc: Wu, Jingjing (jingjing.wu@intel.com); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-24 11:24 ` Hanoch Haim (hhaim)
@ 2017-11-28 12:50 ` Wu, Jingjing
2017-11-28 13:14 ` Hanoch Haim (hhaim)
0 siblings, 1 reply; 12+ messages in thread
From: Wu, Jingjing @ 2017-11-28 12:50 UTC (permalink / raw)
To: Hanoch Haim (hhaim), dev
Hi, Hanoch
Are you talking about i40 VF's latency? And you are using DPDK PF as host driver?
In this case, we are setting the Interrupt Throttling (ITR) to be maximum. That is to say, if the packet rate is very slow , the receive descriptor is written back when ITR timeout, otherwise it is written back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
If we change the ITR to minor, then huge number of interrupts will coming to core which impact performance.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, November 24, 2017 7:25 PM
To: dev@dpdk.org
Cc: Wu, Jingjing <jingjing.wu@intel.com>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-28 12:50 ` Wu, Jingjing
@ 2017-11-28 13:14 ` Hanoch Haim (hhaim)
2017-11-28 15:01 ` Wu, Jingjing
0 siblings, 1 reply; 12+ messages in thread
From: Hanoch Haim (hhaim) @ 2017-11-28 13:14 UTC (permalink / raw)
To: Wu, Jingjing; +Cc: dev, Hanoch Haim (hhaim)
Hi Jingjing,
1. The issue is with DPDK PF.
2. The rate is high ~10gb, one DP core, one latency core.
3. The fix is here
/* Bind all RX queues to allocated MSIX interrupt */
for (i = 0; i < nb_queue; i++) {
val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
#ifdef TREX_PATCH
I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
#else
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
#endif
I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
The Interrupt Throttling ITR is configure using a different setting using a different register here :
4. The ITR_INTERVAL is 32 usec and it affect a different PF register
#define I40E_ITR_INDEX_DEFAULT 0
#define I40E_ITR_INDEX_NONE 3
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
5. My question is why the VF configuration affects PF INT_INTERVAL ? Can I remove my patch and fix this latency issue in the different way?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 2:50 PM
To: Hanoch Haim (hhaim); dev@dpdk.org
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
Are you talking about i40 VF's latency? And you are using DPDK PF as host driver?
In this case, we are setting the Interrupt Throttling (ITR) to be maximum. That is to say, if the packet rate is very slow , the receive descriptor is written back when ITR timeout, otherwise it is written back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
If we change the ITR to minor, then huge number of interrupts will coming to core which impact performance.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, November 24, 2017 7:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-28 13:14 ` Hanoch Haim (hhaim)
@ 2017-11-28 15:01 ` Wu, Jingjing
2017-11-28 15:07 ` Hanoch Haim (hhaim)
2017-11-30 16:47 ` Hanoch Haim (hhaim)
0 siblings, 2 replies; 12+ messages in thread
From: Wu, Jingjing @ 2017-11-28 15:01 UTC (permalink / raw)
To: Hanoch Haim (hhaim); +Cc: dev
Hi, Hanoch
If DPDK PF, the commit affects that because it introduces an argument (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with default value 32us.
If you'd like to get the descriptor write back immediately, you can set "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
Or you can just change the definition of I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Tuesday, November 28, 2017 9:14 PM
To: Wu, Jingjing <jingjing.wu@intel.com>
Cc: dev@dpdk.org; Hanoch Haim (hhaim) <hhaim@cisco.com>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jingjing,
1. The issue is with DPDK PF.
2. The rate is high ~10gb, one DP core, one latency core.
3. The fix is here
/* Bind all RX queues to allocated MSIX interrupt */
for (i = 0; i < nb_queue; i++) {
val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
#ifdef TREX_PATCH
I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
#else
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
#endif
I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
The Interrupt Throttling ITR is configure using a different setting using a different register here :
4. The ITR_INTERVAL is 32 usec and it affect a different PF register
#define I40E_ITR_INDEX_DEFAULT 0
#define I40E_ITR_INDEX_NONE 3
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
5. My question is why the VF configuration affects PF INT_INTERVAL ? Can I remove my patch and fix this latency issue in the different way?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 2:50 PM
To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
Are you talking about i40 VF's latency? And you are using DPDK PF as host driver?
In this case, we are setting the Interrupt Throttling (ITR) to be maximum. That is to say, if the packet rate is very slow , the receive descriptor is written back when ITR timeout, otherwise it is written back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
If we change the ITR to minor, then huge number of interrupts will coming to core which impact performance.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, November 24, 2017 7:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-28 15:01 ` Wu, Jingjing
@ 2017-11-28 15:07 ` Hanoch Haim (hhaim)
2017-11-30 1:24 ` Wu, Jingjing
2017-11-30 16:47 ` Hanoch Haim (hhaim)
1 sibling, 1 reply; 12+ messages in thread
From: Hanoch Haim (hhaim) @ 2017-11-28 15:07 UTC (permalink / raw)
To: Wu, Jingjing; +Cc: dev
Hi Jinjing,
So to verify,
Before the VF commit ("net/i40e: fix interrupt throttling setting in PF") there was a bug that I40E_QUEUE_ITR_INTERVAL_DEFAULT
did *not* work and PF was configured with RX_INT_INTERVAL=0 usec and now it is 32usec (taken from the define)?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 5:01 PM
To: Hanoch Haim (hhaim)
Cc: dev@dpdk.org
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
If DPDK PF, the commit affects that because it introduces an argument (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with default value 32us.
If you'd like to get the descriptor write back immediately, you can set "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
Or you can just change the definition of I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Tuesday, November 28, 2017 9:14 PM
To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim) <hhaim@cisco.com<mailto:hhaim@cisco.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jingjing,
1. The issue is with DPDK PF.
2. The rate is high ~10gb, one DP core, one latency core.
3. The fix is here
/* Bind all RX queues to allocated MSIX interrupt */
for (i = 0; i < nb_queue; i++) {
val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
#ifdef TREX_PATCH
I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
#else
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
#endif
I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
The Interrupt Throttling ITR is configure using a different setting using a different register here :
4. The ITR_INTERVAL is 32 usec and it affect a different PF register
#define I40E_ITR_INDEX_DEFAULT 0
#define I40E_ITR_INDEX_NONE 3
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
5. My question is why the VF configuration affects PF INT_INTERVAL ? Can I remove my patch and fix this latency issue in the different way?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 2:50 PM
To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
Are you talking about i40 VF's latency? And you are using DPDK PF as host driver?
In this case, we are setting the Interrupt Throttling (ITR) to be maximum. That is to say, if the packet rate is very slow , the receive descriptor is written back when ITR timeout, otherwise it is written back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
If we change the ITR to minor, then huge number of interrupts will coming to core which impact performance.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, November 24, 2017 7:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-28 15:07 ` Hanoch Haim (hhaim)
@ 2017-11-30 1:24 ` Wu, Jingjing
0 siblings, 0 replies; 12+ messages in thread
From: Wu, Jingjing @ 2017-11-30 1:24 UTC (permalink / raw)
To: Hanoch Haim (hhaim); +Cc: dev
Hi, Hanoh
Yes, that's true.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Tuesday, November 28, 2017 11:07 PM
To: Wu, Jingjing <jingjing.wu@intel.com>
Cc: dev@dpdk.org
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jinjing,
So to verify,
Before the VF commit ("net/i40e: fix interrupt throttling setting in PF") there was a bug that I40E_QUEUE_ITR_INTERVAL_DEFAULT
did *not* work and PF was configured with RX_INT_INTERVAL=0 usec and now it is 32usec (taken from the define)?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 5:01 PM
To: Hanoch Haim (hhaim)
Cc: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
If DPDK PF, the commit affects that because it introduces an argument (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with default value 32us.
If you'd like to get the descriptor write back immediately, you can set "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
Or you can just change the definition of I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Tuesday, November 28, 2017 9:14 PM
To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim) <hhaim@cisco.com<mailto:hhaim@cisco.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jingjing,
1. The issue is with DPDK PF.
2. The rate is high ~10gb, one DP core, one latency core.
3. The fix is here
/* Bind all RX queues to allocated MSIX interrupt */
for (i = 0; i < nb_queue; i++) {
val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
#ifdef TREX_PATCH
I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
#else
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
#endif
I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
The Interrupt Throttling ITR is configure using a different setting using a different register here :
4. The ITR_INTERVAL is 32 usec and it affect a different PF register
#define I40E_ITR_INDEX_DEFAULT 0
#define I40E_ITR_INDEX_NONE 3
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
5. My question is why the VF configuration affects PF INT_INTERVAL ? Can I remove my patch and fix this latency issue in the different way?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 2:50 PM
To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
Are you talking about i40 VF's latency? And you are using DPDK PF as host driver?
In this case, we are setting the Interrupt Throttling (ITR) to be maximum. That is to say, if the packet rate is very slow , the receive descriptor is written back when ITR timeout, otherwise it is written back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
If we change the ITR to minor, then huge number of interrupts will coming to core which impact performance.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, November 24, 2017 7:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-28 15:01 ` Wu, Jingjing
2017-11-28 15:07 ` Hanoch Haim (hhaim)
@ 2017-11-30 16:47 ` Hanoch Haim (hhaim)
2017-11-30 17:46 ` Wu, Jingjing
1 sibling, 1 reply; 12+ messages in thread
From: Hanoch Haim (hhaim) @ 2017-11-30 16:47 UTC (permalink / raw)
To: Wu, Jingjing; +Cc: dev, Hanoch Haim (hhaim)
Hi Jingjing,
I did that and see the results, It does not work as expected
TRex command:
$sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c 1
1) with the issue (without the patch)
*itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
*#define RTE_LIBRTE_I40E_ITR_INTERVAL -1
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 41 41 34 40 41 33 43 8 8
1 | 5603, 5603, 0, 0, 12 , 44, 24 | 0 0 0 0 39 41 41 34 40 33 33 8 8
2 | 5603, 5603, 0, 0, 8 , 43, 5 | 0 0 0 0 38 41 42 40 40 40 42 9 8
3 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 36 41 34 42 43 8 35 40 41
*** TRex is shutting down - cause: 'CTRL + C detected'
2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
#define RTE_LIBRTE_I40E_ITR_INTERVAL 4
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24 32 23 24 23
1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19 20 30 18 21
2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40 41 41 40 45
3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44 43 43 44 48
*
3) RTE_LIBRTE_I40E_ITR_INTERVAL 0
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
#define RTE_LIBRTE_I40E_ITR_INTERVAL 0
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24 32 23 24 23
1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19 20 30 18 21
2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40 41 41 40 45
3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44 43 43 44 48
*
4) TRex patch issue solved
I40E_QINT_RQCTL_ITR_INDX_MASK
#define RTE_LIBRTE_I40E_ITR_INTERVAL -1
latency : 8usec
-Latency stats enabled
Cpu Utilization : 0.1 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 9501, 9501, 0, 0, 7 , 21, 0 | 21 9 9 12 9 9 14 12 8 9 9 9 8
1 | 9501, 9501, 0, 0, 7 , 25, 0 | 22 8 9 12 9 9 15 12 8 8 9 9 8
2 | 9501, 9501, 0, 0, 6 , 26, 0 | 22 9 10 9 10 10 15 13 8 9 9 9 8
3 | 9501, 9501, 0, 0, 6 , 32, 0 | 22 8 9 9 9 9 15 12 8 7 9 9 9
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 5:01 PM
To: Hanoch Haim (hhaim)
Cc: dev@dpdk.org
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
If DPDK PF, the commit affects that because it introduces an argument (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with default value 32us.
If you'd like to get the descriptor write back immediately, you can set "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
Or you can just change the definition of I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Tuesday, November 28, 2017 9:14 PM
To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim) <hhaim@cisco.com<mailto:hhaim@cisco.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jingjing,
1. The issue is with DPDK PF.
2. The rate is high ~10gb, one DP core, one latency core.
3. The fix is here
/* Bind all RX queues to allocated MSIX interrupt */
for (i = 0; i < nb_queue; i++) {
val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
#ifdef TREX_PATCH
I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
#else
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
#endif
I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
The Interrupt Throttling ITR is configure using a different setting using a different register here :
4. The ITR_INTERVAL is 32 usec and it affect a different PF register
#define I40E_ITR_INDEX_DEFAULT 0
#define I40E_ITR_INDEX_NONE 3
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
5. My question is why the VF configuration affects PF INT_INTERVAL ? Can I remove my patch and fix this latency issue in the different way?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 2:50 PM
To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
Are you talking about i40 VF's latency? And you are using DPDK PF as host driver?
In this case, we are setting the Interrupt Throttling (ITR) to be maximum. That is to say, if the packet rate is very slow , the receive descriptor is written back when ITR timeout, otherwise it is written back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
If we change the ITR to minor, then huge number of interrupts will coming to core which impact performance.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, November 24, 2017 7:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-30 16:47 ` Hanoch Haim (hhaim)
@ 2017-11-30 17:46 ` Wu, Jingjing
2017-11-30 18:02 ` Hanoch Haim (hhaim)
2017-12-06 11:00 ` Xing, Beilei
0 siblings, 2 replies; 12+ messages in thread
From: Wu, Jingjing @ 2017-11-30 17:46 UTC (permalink / raw)
To: Hanoch Haim (hhaim); +Cc: dev
Hi, Hanoch
Thanks a lot for the trying. Are you using igb_uio to bind the device?
I guess it would be because that fix is not complete, the overlap with Rx interrupt mode is not considered. We will look into it.
And it would be great if you can have a try on vfio_pci cases.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, December 1, 2017 12:48 AM
To: Wu, Jingjing <jingjing.wu@intel.com>
Cc: dev@dpdk.org; Hanoch Haim (hhaim) <hhaim@cisco.com>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jingjing,
I did that and see the results, It does not work as expected
TRex command:
$sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c 1
1) with the issue (without the patch)
*itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
*#define RTE_LIBRTE_I40E_ITR_INTERVAL -1
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 41 41 34 40 41 33 43 8 8
1 | 5603, 5603, 0, 0, 12 , 44, 24 | 0 0 0 0 39 41 41 34 40 33 33 8 8
2 | 5603, 5603, 0, 0, 8 , 43, 5 | 0 0 0 0 38 41 42 40 40 40 42 9 8
3 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 36 41 34 42 43 8 35 40 41
*** TRex is shutting down - cause: 'CTRL + C detected'
2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
#define RTE_LIBRTE_I40E_ITR_INTERVAL 4
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24 32 23 24 23
1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19 20 30 18 21
2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40 41 41 40 45
3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44 43 43 44 48
*
3) RTE_LIBRTE_I40E_ITR_INTERVAL 0
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
#define RTE_LIBRTE_I40E_ITR_INTERVAL 0
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24 32 23 24 23
1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19 20 30 18 21
2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40 41 41 40 45
3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44 43 43 44 48
*
4) TRex patch issue solved
I40E_QINT_RQCTL_ITR_INDX_MASK
#define RTE_LIBRTE_I40E_ITR_INTERVAL -1
latency : 8usec
-Latency stats enabled
Cpu Utilization : 0.1 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 9501, 9501, 0, 0, 7 , 21, 0 | 21 9 9 12 9 9 14 12 8 9 9 9 8
1 | 9501, 9501, 0, 0, 7 , 25, 0 | 22 8 9 12 9 9 15 12 8 8 9 9 8
2 | 9501, 9501, 0, 0, 6 , 26, 0 | 22 9 10 9 10 10 15 13 8 9 9 9 8
3 | 9501, 9501, 0, 0, 6 , 32, 0 | 22 8 9 9 9 9 15 12 8 7 9 9 9
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 5:01 PM
To: Hanoch Haim (hhaim)
Cc: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
If DPDK PF, the commit affects that because it introduces an argument (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with default value 32us.
If you'd like to get the descriptor write back immediately, you can set "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
Or you can just change the definition of I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Tuesday, November 28, 2017 9:14 PM
To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim) <hhaim@cisco.com<mailto:hhaim@cisco.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jingjing,
1. The issue is with DPDK PF.
2. The rate is high ~10gb, one DP core, one latency core.
3. The fix is here
/* Bind all RX queues to allocated MSIX interrupt */
for (i = 0; i < nb_queue; i++) {
val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
#ifdef TREX_PATCH
I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
#else
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
#endif
I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
The Interrupt Throttling ITR is configure using a different setting using a different register here :
4. The ITR_INTERVAL is 32 usec and it affect a different PF register
#define I40E_ITR_INDEX_DEFAULT 0
#define I40E_ITR_INDEX_NONE 3
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
5. My question is why the VF configuration affects PF INT_INTERVAL ? Can I remove my patch and fix this latency issue in the different way?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 2:50 PM
To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
Are you talking about i40 VF's latency? And you are using DPDK PF as host driver?
In this case, we are setting the Interrupt Throttling (ITR) to be maximum. That is to say, if the packet rate is very slow , the receive descriptor is written back when ITR timeout, otherwise it is written back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
If we change the ITR to minor, then huge number of interrupts will coming to core which impact performance.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, November 24, 2017 7:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-30 17:46 ` Wu, Jingjing
@ 2017-11-30 18:02 ` Hanoch Haim (hhaim)
2017-12-06 11:00 ` Xing, Beilei
1 sibling, 0 replies; 12+ messages in thread
From: Hanoch Haim (hhaim) @ 2017-11-30 18:02 UTC (permalink / raw)
To: Wu, Jingjing; +Cc: dev
Igb_uio. I will try vfio.
Is there a better workaround to force low_latency in Rx side (without our patch)?
Hanoh
Sent from my iPhone
On 30 Nov 2017, at 19:47, Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>> wrote:
Hi, Hanoch
Thanks a lot for the trying. Are you using igb_uio to bind the device?
I guess it would be because that fix is not complete, the overlap with Rx interrupt mode is not considered. We will look into it.
And it would be great if you can have a try on vfio_pci cases.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, December 1, 2017 12:48 AM
To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim) <hhaim@cisco.com<mailto:hhaim@cisco.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jingjing,
I did that and see the results, It does not work as expected
TRex command:
$sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c 1
1) with the issue (without the patch)
*itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
*#define RTE_LIBRTE_I40E_ITR_INTERVAL -1
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 41 41 34 40 41 33 43 8 8
1 | 5603, 5603, 0, 0, 12 , 44, 24 | 0 0 0 0 39 41 41 34 40 33 33 8 8
2 | 5603, 5603, 0, 0, 8 , 43, 5 | 0 0 0 0 38 41 42 40 40 40 42 9 8
3 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 36 41 34 42 43 8 35 40 41
*** TRex is shutting down - cause: 'CTRL + C detected'
2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
#define RTE_LIBRTE_I40E_ITR_INTERVAL 4
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24 32 23 24 23
1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19 20 30 18 21
2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40 41 41 40 45
3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44 43 43 44 48
*
3) RTE_LIBRTE_I40E_ITR_INTERVAL 0
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
#define RTE_LIBRTE_I40E_ITR_INTERVAL 0
latency : 40usec
-Latency stats enabled
Cpu Utilization : 0.2 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24 32 23 24 23
1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19 20 30 18 21
2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40 41 41 40 45
3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44 43 43 44 48
*
4) TRex patch issue solved
I40E_QINT_RQCTL_ITR_INDX_MASK
#define RTE_LIBRTE_I40E_ITR_INTERVAL -1
latency : 8usec
-Latency stats enabled
Cpu Utilization : 0.1 %
if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window
| , , , , average , max , (usec)
----------------------------------------------------------------------------------------------------------------
0 | 9501, 9501, 0, 0, 7 , 21, 0 | 21 9 9 12 9 9 14 12 8 9 9 9 8
1 | 9501, 9501, 0, 0, 7 , 25, 0 | 22 8 9 12 9 9 15 12 8 8 9 9 8
2 | 9501, 9501, 0, 0, 6 , 26, 0 | 22 9 10 9 10 10 15 13 8 9 9 9 8
3 | 9501, 9501, 0, 0, 6 , 32, 0 | 22 8 9 9 9 9 15 12 8 7 9 9 9
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 5:01 PM
To: Hanoch Haim (hhaim)
Cc: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
If DPDK PF, the commit affects that because it introduces an argument (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with default value 32us.
If you’d like to get the descriptor write back immediately, you can set “CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0” in config/common_base file.
Or you can just change the definition of I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Tuesday, November 28, 2017 9:14 PM
To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim) <hhaim@cisco.com<mailto:hhaim@cisco.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi Jingjing,
1. The issue is with DPDK PF.
2. The rate is high ~10gb, one DP core, one latency core.
3. The fix is here
/* Bind all RX queues to allocated MSIX interrupt */
for (i = 0; i < nb_queue; i++) {
val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
#ifdef TREX_PATCH
I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
#else
itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
#endif
I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
The Interrupt Throttling ITR is configure using a different setting using a different register here :
4. The ITR_INTERVAL is 32 usec and it affect a different PF register
#define I40E_ITR_INDEX_DEFAULT 0
#define I40E_ITR_INDEX_NONE 3
#define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
5. My question is why the VF configuration affects PF INT_INTERVAL ? Can I remove my patch and fix this latency issue in the different way?
Thanks,
Hanoh
From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
Sent: Tuesday, November 28, 2017 2:50 PM
To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi, Hanoch
Are you talking about i40 VF’s latency? And you are using DPDK PF as host driver?
In this case, we are setting the Interrupt Throttling (ITR) to be maximum. That is to say, if the packet rate is very slow , the receive descriptor is written back when ITR timeout, otherwise it is written back when cache line is full (4 descriptors/packets). I think that’s why you saw the latency is varying.
If we change the ITR to minor, then huge number of interrupts will coming to core which impact performance.
Thanks
Jingjing
From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
Sent: Friday, November 24, 2017 7:25 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Re-sending
Hanoh
From: Hanoch Haim (hhaim)
Sent: Monday, November 20, 2017 5:19 PM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>); Hanoch Haim (hhaim)
Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi All,
While integrating dpdk17.11 into TRex latest code a new latency issue is observed (i40e is very sensitive because it has very good resolution due to Qos configuration).
git bitsec found the following commit.
With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
Any idea why?
I can send how to reproduce this, it is very simple.
cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Date: Thu Aug 24 09:57:51 2017 +0800
net/i40e: fix interrupt throttling setting in PF
As no matter the PF host driver is DPDK or other kernel drivers,
they are sharing the same virtchnnl interfaces to communicate to VFs.
To follow the generic interface, DPDK PF need to set Interrupt
Throttling (ITR) index according to the rxitr_idx from virtchnnl
instead of ITR_NONE.
Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux VF")
Cc: stable@dpdk.org<mailto:stable@dpdk.org>
Signed-off-by: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
Thanks,
Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-11-30 17:46 ` Wu, Jingjing
2017-11-30 18:02 ` Hanoch Haim (hhaim)
@ 2017-12-06 11:00 ` Xing, Beilei
2017-12-06 11:21 ` Hanoch Haim (hhaim)
1 sibling, 1 reply; 12+ messages in thread
From: Xing, Beilei @ 2017-12-06 11:00 UTC (permalink / raw)
To: Wu, Jingjing, Hanoch Haim (hhaim); +Cc: dev
Hi,
I tested Rx latency with testpmd and ixia, found that the interval configuration works on X710.
With the default configuration, the latency is about 32us.
When RTE_LIBRTE_I40E_ITR_INTERVAL=0, the max latency is < 8us.
When RTE_LIBRTE_I40E_ITR_INTERVAL=8160, the max latency is about 8ms.
My test steps:
1. connect a X710 port with ixia, and bind the port to igb_uio
2. run testpmd and start io forwding
3. send a burst (5 packets) with ixia to X710 port
4. check the latency on ixia.
Best Regards,
Beilei
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wu, Jingjing
> Sent: Friday, December 1, 2017 1:47 AM
> To: Hanoch Haim (hhaim) <hhaim@cisco.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
>
> Hi, Hanoch
>
> Thanks a lot for the trying. Are you using igb_uio to bind the device?
>
> I guess it would be because that fix is not complete, the overlap with Rx
> interrupt mode is not considered. We will look into it.
> And it would be great if you can have a try on vfio_pci cases.
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Friday, December 1, 2017 12:48 AM
> To: Wu, Jingjing <jingjing.wu@intel.com>
> Cc: dev@dpdk.org; Hanoch Haim (hhaim) <hhaim@cisco.com>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
>
> Hi Jingjing,
> I did that and see the results, It does not work as expected
>
> TRex command:
> $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c 1
>
>
>
> 1) with the issue (without the patch)
> *itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | *#define
> RTE_LIBRTE_I40E_ITR_INTERVAL -1
>
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
> ---------------------------------------------------------------------------------------------
> -------------------
> 0 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 41 41 34 40 41
> 33 43 8 8
> 1 | 5603, 5603, 0, 0, 12 , 44, 24 | 0 0 0 0 39 41 41 34 40
> 33 33 8 8
> 2 | 5603, 5603, 0, 0, 8 , 43, 5 | 0 0 0 0 38 41 42 40 40
> 40 42 9 8
> 3 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 36 41 34 42 43
> 8 35 40 41
> *** TRex is shutting down - cause: 'CTRL + C detected'
>
> 2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4
>
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define
> RTE_LIBRTE_I40E_ITR_INTERVAL 4
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
> ---------------------------------------------------------------------------------------------
> -------------------
> 0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24
> 32 23 24 23
> 1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19
> 20 30 18 21
> 2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40
> 41 41 40 45
> 3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44
> 43 43 44 48
> *
>
>
> 3) RTE_LIBRTE_I40E_ITR_INTERVAL 0
>
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define
> RTE_LIBRTE_I40E_ITR_INTERVAL 0
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
> ---------------------------------------------------------------------------------------------
> -------------------
> 0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24
> 32 23 24 23
> 1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19
> 20 30 18 21
> 2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40
> 41 41 40 45
> 3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44
> 43 43 44 48
> *
>
> 4) TRex patch issue solved
>
> I40E_QINT_RQCTL_ITR_INDX_MASK
> #define RTE_LIBRTE_I40E_ITR_INTERVAL -1
>
> latency : 8usec
>
> -Latency stats enabled
> Cpu Utilization : 0.1 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
> ---------------------------------------------------------------------------------------------
> -------------------
> 0 | 9501, 9501, 0, 0, 7 , 21, 0 | 21 9 9 12 9 9 14 12 8 9
> 9 9 8
> 1 | 9501, 9501, 0, 0, 7 , 25, 0 | 22 8 9 12 9 9 15 12 8 8
> 9 9 8
> 2 | 9501, 9501, 0, 0, 6 , 26, 0 | 22 9 10 9 10 10 15 13 8
> 9 9 9 8
> 3 | 9501, 9501, 0, 0, 6 , 32, 0 | 22 8 9 9 9 9 15 12 8 7
> 9 9 9
>
> Thanks,
> Hanoh
>
> From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
> Sent: Tuesday, November 28, 2017 5:01 PM
> To: Hanoch Haim (hhaim)
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
>
> Hi, Hanoch
>
> If DPDK PF, the commit affects that because it introduces an argument
> (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with
> default value 32us.
>
> If you'd like to get the descriptor write back immediately, you can set
> "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
>
> Or you can just change the definition of
> I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Tuesday, November 28, 2017 9:14 PM
> To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim)
> <hhaim@cisco.com<mailto:hhaim@cisco.com>>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
>
> Hi Jingjing,
>
> 1. The issue is with DPDK PF.
>
> 2. The rate is high ~10gb, one DP core, one latency core.
>
> 3. The fix is here
>
> /* Bind all RX queues to allocated MSIX interrupt */
> for (i = 0; i < nb_queue; i++) {
> val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
> #ifdef TREX_PATCH
> I40E_QINT_RQCTL_ITR_INDX_MASK |
> << low latency 11b = NoITR
> #else
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
> << high spkies
> #endif
>
> I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
>
> The Interrupt Throttling ITR is configure using a different setting using a
> different register here :
>
>
> 4. The ITR_INTERVAL is 32 usec and it affect a different PF register
>
> #define I40E_ITR_INDEX_DEFAULT 0
> #define I40E_ITR_INDEX_NONE 3
> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
>
>
> 5. My question is why the VF configuration affects PF INT_INTERVAL ? Can
> I remove my patch and fix this latency issue in the different way?
>
> Thanks,
> Hanoh
>
> From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
> Sent: Tuesday, November 28, 2017 2:50 PM
> To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
>
> Hi, Hanoch
>
> Are you talking about i40 VF's latency? And you are using DPDK PF as host
> driver?
>
> In this case, we are setting the Interrupt Throttling (ITR) to be maximum.
> That is to say, if the packet rate is very slow , the receive descriptor is written
> back when ITR timeout, otherwise it is written back when cache line is full (4
> descriptors/packets). I think that's why you saw the latency is varying.
>
> If we change the ITR to minor, then huge number of interrupts will coming to
> core which impact performance.
>
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Friday, November 24, 2017 7:25 PM
> To: dev@dpdk.org<mailto:dev@dpdk.org>
> Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
>
> Re-sending
>
> Hanoh
>
> From: Hanoch Haim (hhaim)
> Sent: Monday, November 20, 2017 5:19 PM
> To: dev@dpdk.org<mailto:dev@dpdk.org>
> Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>);
> Hanoch Haim (hhaim)
> Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting
> in PF
>
> Hi All,
> While integrating dpdk17.11 into TRex latest code a new latency issue is
> observed (i40e is very sensitive because it has very good resolution due to
> Qos configuration).
> git bitsec found the following commit.
> With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
> Any idea why?
> I can send how to reproduce this, it is very simple.
>
> cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
> commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
> Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Date: Thu Aug 24 09:57:51 2017 +0800
>
> net/i40e: fix interrupt throttling setting in PF
>
> As no matter the PF host driver is DPDK or other kernel drivers,
> they are sharing the same virtchnnl interfaces to communicate to VFs.
> To follow the generic interface, DPDK PF need to set Interrupt
> Throttling (ITR) index according to the rxitr_idx from virtchnnl
> instead of ITR_NONE.
>
> Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux
> VF")
> Cc: stable@dpdk.org<mailto:stable@dpdk.org>
>
> Signed-off-by: Jingjing Wu
> <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
>
>
>
> Thanks,
> Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: net/i40e: latency issue due fix interrupt throttling setting in PF
2017-12-06 11:00 ` Xing, Beilei
@ 2017-12-06 11:21 ` Hanoch Haim (hhaim)
0 siblings, 0 replies; 12+ messages in thread
From: Hanoch Haim (hhaim) @ 2017-12-06 11:21 UTC (permalink / raw)
To: Xing, Beilei, Wu, Jingjing; +Cc: dev
Hi Beilei,
Thanks for looking into this.
I'm testing it with TRex (tx->rx) and not (rx->tx) in a bit different configuration and there are packets with ~40usec latency (Tx/Rx) without my patch
So I assume the latency comes from a different place due to this change (INTR)
TRex command:
$sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c 1
I'm testing it in loopback and in this configuration there are two Tx queues and 2 Rx queues
RSS is not configured all Rx traffic goes to Rx0
Core0
--------
Tx0 - TCP flows
Rx0 - TCP flows
Core1
--------
Tx1 - latency flows (ICMP) - low rate 1KPPS
Rx1 - latency flows (ICMP) - match a rule of TOS/TTL to redirect to this Rx1 queue.
Qos configuration
------------------------
ETS, Tx1 is configured as low latency/high priority
Throughput of Tx0/Rx0 is ~10Gb/average packet size =~700B
The latency is measured betwean Tx1 to Rx1
Thanks,
Hanoh
-----Original Message-----
From: Xing, Beilei [mailto:beilei.xing@intel.com]
Sent: Wednesday, December 06, 2017 1:00 PM
To: Wu, Jingjing; Hanoch Haim (hhaim)
Cc: dev@dpdk.org
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi,
I tested Rx latency with testpmd and ixia, found that the interval configuration works on X710.
With the default configuration, the latency is about 32us.
When RTE_LIBRTE_I40E_ITR_INTERVAL=0, the max latency is < 8us.
When RTE_LIBRTE_I40E_ITR_INTERVAL=8160, the max latency is about 8ms.
My test steps:
1. connect a X710 port with ixia, and bind the port to igb_uio 2. run testpmd and start io forwding 3. send a burst (5 packets) with ixia to X710 port 4. check the latency on ixia.
Best Regards,
Beilei
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wu, Jingjing
> Sent: Friday, December 1, 2017 1:47 AM
> To: Hanoch Haim (hhaim) <hhaim@cisco.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi, Hanoch
>
> Thanks a lot for the trying. Are you using igb_uio to bind the device?
>
> I guess it would be because that fix is not complete, the overlap with
> Rx interrupt mode is not considered. We will look into it.
> And it would be great if you can have a try on vfio_pci cases.
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Friday, December 1, 2017 12:48 AM
> To: Wu, Jingjing <jingjing.wu@intel.com>
> Cc: dev@dpdk.org; Hanoch Haim (hhaim) <hhaim@cisco.com>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi Jingjing,
> I did that and see the results, It does not work as expected
>
> TRex command:
> $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000
> --astf -c 1
>
>
>
> 1) with the issue (without the patch)
> *itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | *#define
> RTE_LIBRTE_I40E_ITR_INTERVAL -1
>
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
>
> ----------------------------------------------------------------------
> -----------------------
> -------------------
> 0 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 41 41 34 40 41
> 33 43 8 8
> 1 | 5603, 5603, 0, 0, 12 , 44, 24 | 0 0 0 0 39 41 41 34 40
> 33 33 8 8
> 2 | 5603, 5603, 0, 0, 8 , 43, 5 | 0 0 0 0 38 41 42 40 40
> 40 42 9 8
> 3 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 36 41 34 42 43
> 8 35 40 41
> *** TRex is shutting down - cause: 'CTRL + C detected'
>
> 2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4
>
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define
> RTE_LIBRTE_I40E_ITR_INTERVAL 4
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
>
> ----------------------------------------------------------------------
> -----------------------
> -------------------
> 0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24
> 32 23 24 23
> 1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19
> 20 30 18 21
> 2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40
> 41 41 40 45
> 3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44
> 43 43 44 48
> *
>
>
> 3) RTE_LIBRTE_I40E_ITR_INTERVAL 0
>
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define
> RTE_LIBRTE_I40E_ITR_INTERVAL 0
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
>
> ----------------------------------------------------------------------
> -----------------------
> -------------------
> 0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24
> 32 23 24 23
> 1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19
> 20 30 18 21
> 2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40
> 41 41 40 45
> 3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44
> 43 43 44 48
> *
>
> 4) TRex patch issue solved
>
> I40E_QINT_RQCTL_ITR_INDX_MASK
> #define RTE_LIBRTE_I40E_ITR_INTERVAL -1
>
> latency : 8usec
>
> -Latency stats enabled
> Cpu Utilization : 0.1 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
>
> ----------------------------------------------------------------------
> -----------------------
> -------------------
> 0 | 9501, 9501, 0, 0, 7 , 21, 0 | 21 9 9 12 9 9 14 12 8 9
> 9 9 8
> 1 | 9501, 9501, 0, 0, 7 , 25, 0 | 22 8 9 12 9 9 15 12 8 8
> 9 9 8
> 2 | 9501, 9501, 0, 0, 6 , 26, 0 | 22 9 10 9 10 10 15 13 8
> 9 9 9 8
> 3 | 9501, 9501, 0, 0, 6 , 32, 0 | 22 8 9 9 9 9 15 12 8 7
> 9 9 9
>
> Thanks,
> Hanoh
>
> From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
> Sent: Tuesday, November 28, 2017 5:01 PM
> To: Hanoch Haim (hhaim)
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi, Hanoch
>
> If DPDK PF, the commit affects that because it introduces an argument
> (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx
> with default value 32us.
>
> If you'd like to get the descriptor write back immediately, you can
> set "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
>
> Or you can just change the definition of
> I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Tuesday, November 28, 2017 9:14 PM
> To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim)
> <hhaim@cisco.com<mailto:hhaim@cisco.com>>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi Jingjing,
>
> 1. The issue is with DPDK PF.
>
> 2. The rate is high ~10gb, one DP core, one latency core.
>
> 3. The fix is here
>
> /* Bind all RX queues to allocated MSIX interrupt */
> for (i = 0; i < nb_queue; i++) {
> val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
> #ifdef TREX_PATCH
>
> I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
> #else
> itr_idx <<
> I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
> #endif
>
> I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
>
> The Interrupt Throttling ITR is configure using a different setting
> using a different register here :
>
>
> 4. The ITR_INTERVAL is 32 usec and it affect a different PF register
>
> #define I40E_ITR_INDEX_DEFAULT 0
> #define I40E_ITR_INDEX_NONE 3
> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
>
>
> 5. My question is why the VF configuration affects PF INT_INTERVAL ? Can
> I remove my patch and fix this latency issue in the different way?
>
> Thanks,
> Hanoh
>
> From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
> Sent: Tuesday, November 28, 2017 2:50 PM
> To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi, Hanoch
>
> Are you talking about i40 VF's latency? And you are using DPDK PF as
> host driver?
>
> In this case, we are setting the Interrupt Throttling (ITR) to be maximum.
> That is to say, if the packet rate is very slow , the receive
> descriptor is written back when ITR timeout, otherwise it is written
> back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
>
> If we change the ITR to minor, then huge number of interrupts will
> coming to core which impact performance.
>
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Friday, November 24, 2017 7:25 PM
> To: dev@dpdk.org<mailto:dev@dpdk.org>
> Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Re-sending
>
> Hanoh
>
> From: Hanoch Haim (hhaim)
> Sent: Monday, November 20, 2017 5:19 PM
> To: dev@dpdk.org<mailto:dev@dpdk.org>
> Cc: Wu, Jingjing
> (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>);
> Hanoch Haim (hhaim)
> Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi All,
> While integrating dpdk17.11 into TRex latest code a new latency issue
> is observed (i40e is very sensitive because it has very good
> resolution due to Qos configuration).
> git bitsec found the following commit.
> With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
> Any idea why?
> I can send how to reproduce this, it is very simple.
>
> cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
> commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
> Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Date: Thu Aug 24 09:57:51 2017 +0800
>
> net/i40e: fix interrupt throttling setting in PF
>
> As no matter the PF host driver is DPDK or other kernel drivers,
> they are sharing the same virtchnnl interfaces to communicate to VFs.
> To follow the generic interface, DPDK PF need to set Interrupt
> Throttling (ITR) index according to the rxitr_idx from virtchnnl
> instead of ITR_NONE.
>
> Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support
> Linux
> VF")
> Cc: stable@dpdk.org<mailto:stable@dpdk.org>
>
> Signed-off-by: Jingjing Wu
> <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
>
>
>
> Thanks,
> Hanoh
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2017-12-06 11:21 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-20 15:18 net/i40e: latency issue due fix interrupt throttling setting in PF Hanoch Haim (hhaim)
2017-11-24 11:24 ` Hanoch Haim (hhaim)
2017-11-28 12:50 ` Wu, Jingjing
2017-11-28 13:14 ` Hanoch Haim (hhaim)
2017-11-28 15:01 ` Wu, Jingjing
2017-11-28 15:07 ` Hanoch Haim (hhaim)
2017-11-30 1:24 ` Wu, Jingjing
2017-11-30 16:47 ` Hanoch Haim (hhaim)
2017-11-30 17:46 ` Wu, Jingjing
2017-11-30 18:02 ` Hanoch Haim (hhaim)
2017-12-06 11:00 ` Xing, Beilei
2017-12-06 11:21 ` Hanoch Haim (hhaim)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.