* Rx_missed_errors drops with larger packet size
@ 2016-05-20 9:09 SwamZ
2016-05-20 10:44 ` Bruce Richardson
2016-05-20 16:45 ` Alexander Duyck
0 siblings, 2 replies; 5+ messages in thread
From: SwamZ @ 2016-05-20 9:09 UTC (permalink / raw)
To: dev
Hi,
While doing performance testing with larger packet size (like 4000 bytes),
we are seeing rx_missed_errors on the interface. This issue is not seen
with packet size less than 2000. There were questions asked in this forum
on rx_missed_error, but there was not any conclusion. Please let me know
what could be the reason for these drops.
I tried the following without any luck:
1) Tried different burst size like 16, 32 an 64
2) Tried different number of the rx descriptors like 512, 1024, 2048 and
4096.
Setup and testing details:
CPU Speed : 2.6 GHz
NIC : 82599ES 10-Gigabit
IO Virtualization: SR-IOV
IO and packet processing cores are in the same NUMA.
Packet size : 4000 bytes
Traffic rate: 80 % line rate. If the traffic rate is < ~80% then drops are
not seen.
Application: This is a sample application developed using L3fwd example app.
DPDK version: 1.7
Thanks,
Swamy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Rx_missed_errors drops with larger packet size
2016-05-20 9:09 Rx_missed_errors drops with larger packet size SwamZ
@ 2016-05-20 10:44 ` Bruce Richardson
2016-05-20 16:45 ` Alexander Duyck
1 sibling, 0 replies; 5+ messages in thread
From: Bruce Richardson @ 2016-05-20 10:44 UTC (permalink / raw)
To: SwamZ; +Cc: dev
On Fri, May 20, 2016 at 02:09:02AM -0700, SwamZ wrote:
> Hi,
>
>
> While doing performance testing with larger packet size (like 4000 bytes),
> we are seeing rx_missed_errors on the interface. This issue is not seen
> with packet size less than 2000. There were questions asked in this forum
> on rx_missed_error, but there was not any conclusion. Please let me know
> what could be the reason for these drops.
>
>
> I tried the following without any luck:
>
> 1) Tried different burst size like 16, 32 an 64
>
> 2) Tried different number of the rx descriptors like 512, 1024, 2048 and
> 4096.
>
>
> Setup and testing details:
>
> CPU Speed : 2.6 GHz
>
> NIC : 82599ES 10-Gigabit
>
> IO Virtualization: SR-IOV
>
> IO and packet processing cores are in the same NUMA.
>
> Packet size : 4000 bytes
>
> Traffic rate: 80 % line rate. If the traffic rate is < ~80% then drops are
> not seen.
>
> Application: This is a sample application developed using L3fwd example app.
>
> DPDK version: 1.7
Hi,
Couple of follow up questions:
1. Can you confirm that you are using the standard ~2k buffer size, implying
that you are using the slower scattered RX/TX code paths for the 4k packet size
case? If so, can you try increasing your buffer size to 5k and see what happens?
2. Does the issue still occur on the latest DPDK 16.04 release, or on the latest
code on dpdk.org. Release 1.7 is very old at this stage.
Regards,
/Bruce
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Rx_missed_errors drops with larger packet size
2016-05-20 9:09 Rx_missed_errors drops with larger packet size SwamZ
2016-05-20 10:44 ` Bruce Richardson
@ 2016-05-20 16:45 ` Alexander Duyck
2016-05-20 21:47 ` SwamZ
1 sibling, 1 reply; 5+ messages in thread
From: Alexander Duyck @ 2016-05-20 16:45 UTC (permalink / raw)
To: SwamZ; +Cc: dev
On Fri, May 20, 2016 at 2:09 AM, SwamZ <swamssrk@gmail.com> wrote:
> Hi,
>
>
> While doing performance testing with larger packet size (like 4000 bytes),
> we are seeing rx_missed_errors on the interface. This issue is not seen
> with packet size less than 2000. There were questions asked in this forum
> on rx_missed_error, but there was not any conclusion. Please let me know
> what could be the reason for these drops.
>
>
> I tried the following without any luck:
>
> 1) Tried different burst size like 16, 32 an 64
>
> 2) Tried different number of the rx descriptors like 512, 1024, 2048 and
> 4096.
I've seen issues like this occur on the 82599 using the Linux kernel
diver in the past, but it was usually related to CPU C states. I'm
assuming that shouldn't apply since you are using the poll mode driver
right so there are no interrupts and the CPUs are busy polling.
Another thing to check is that you have sufficient PCIe bandwidth.
Take a look at an lspci -vvv for the device and make certain it is
connected at x8 Gen2.
> Setup and testing details:
>
> CPU Speed : 2.6 GHz
>
> NIC : 82599ES 10-Gigabit
>
> IO Virtualization: SR-IOV
>
> IO and packet processing cores are in the same NUMA.
If you are running SR-IOV then the SRRCTL_DROP_EN bit should be set in
all the SRRCTL registers if I am not mistaken. As such if you are
triggering Rx missed errors it may not be related to the driver
directly but could point to some sort of PCIe but utilization issue.
How many VFs are you using and is any of this traffic multicast or
broadcast? Multicast or broadcast traffic can potentially saturate
the PCIe bus if there is enough of it on the wire as each packet is
normally replicated for each active VF. So 1 packet on the wire (4K)
can become 8 on the PCIe bus (32K) if you have 7 active VFs.
> Packet size : 4000 bytes
>
> Traffic rate: 80 % line rate. If the traffic rate is < ~80% then drops are
> not seen.
>
> Application: This is a sample application developed using L3fwd example app.
>
> DPDK version: 1.7
> Thanks,
> Swamy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Rx_missed_errors drops with larger packet size
2016-05-20 16:45 ` Alexander Duyck
@ 2016-05-20 21:47 ` SwamZ
2016-05-31 18:50 ` SwamZ
0 siblings, 1 reply; 5+ messages in thread
From: SwamZ @ 2016-05-20 21:47 UTC (permalink / raw)
To: Alexander Duyck; +Cc: dev
Hi Alex and Bruce,
Thanks for the reply.
Alex,
Just saw your email. You are right, this issue was happening for multicast
packets only. On 82599 NIC, multicast and broadcast packet replication
(IXGBE_VT_CTL_REPLEN) was enabled even for single VF case, hence NIC was
dropping the packets as rx_missed_errors. After disabling the replication,
we are not seeing any drops.
Regards,
Swamy SRK
On Fri, May 20, 2016 at 9:45 AM, Alexander Duyck <alexander.duyck@gmail.com>
wrote:
> On Fri, May 20, 2016 at 2:09 AM, SwamZ <swamssrk@gmail.com> wrote:
> > Hi,
> >
> >
> > While doing performance testing with larger packet size (like 4000
> bytes),
> > we are seeing rx_missed_errors on the interface. This issue is not seen
> > with packet size less than 2000. There were questions asked in this forum
> > on rx_missed_error, but there was not any conclusion. Please let me know
> > what could be the reason for these drops.
> >
> >
> > I tried the following without any luck:
> >
> > 1) Tried different burst size like 16, 32 an 64
> >
> > 2) Tried different number of the rx descriptors like 512, 1024, 2048 and
> > 4096.
>
> I've seen issues like this occur on the 82599 using the Linux kernel
> diver in the past, but it was usually related to CPU C states. I'm
> assuming that shouldn't apply since you are using the poll mode driver
> right so there are no interrupts and the CPUs are busy polling.
> Another thing to check is that you have sufficient PCIe bandwidth.
> Take a look at an lspci -vvv for the device and make certain it is
> connected at x8 Gen2.
>
> > Setup and testing details:
> >
> > CPU Speed : 2.6 GHz
> >
> > NIC : 82599ES 10-Gigabit
> >
> > IO Virtualization: SR-IOV
> >
> > IO and packet processing cores are in the same NUMA.
>
> If you are running SR-IOV then the SRRCTL_DROP_EN bit should be set in
> all the SRRCTL registers if I am not mistaken. As such if you are
> triggering Rx missed errors it may not be related to the driver
> directly but could point to some sort of PCIe but utilization issue.
> How many VFs are you using and is any of this traffic multicast or
> broadcast? Multicast or broadcast traffic can potentially saturate
> the PCIe bus if there is enough of it on the wire as each packet is
> normally replicated for each active VF. So 1 packet on the wire (4K)
> can become 8 on the PCIe bus (32K) if you have 7 active VFs.
>
> > Packet size : 4000 bytes
> >
> > Traffic rate: 80 % line rate. If the traffic rate is < ~80% then drops
> are
> > not seen.
> >
> > Application: This is a sample application developed using L3fwd example
> app.
> >
> > DPDK version: 1.7
> > Thanks,
> > Swamy
>
--
Swamz
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Rx_missed_errors drops with larger packet size
2016-05-20 21:47 ` SwamZ
@ 2016-05-31 18:50 ` SwamZ
0 siblings, 0 replies; 5+ messages in thread
From: SwamZ @ 2016-05-31 18:50 UTC (permalink / raw)
To: Alexander Duyck; +Cc: dev
[-- Attachment #1: Type: text/plain, Size: 3618 bytes --]
Hi Alex et al.,
I am seeing another case of rx_missed_error. In this case,
rx_missed_errors are seen only when I send multiple flows. Rx_missed_errors
are NOT seen if I send single flow. Following are the test details:
Packet type: Unicast/Mulitcast
Packet size : 512Bytes
Number of flows: 1000
Rate : 24K PPS (98.3Mb) bidirectional traffic.
CPU Speed : 2.6 GHz
NIC : 82599ES 10-Gigabit
IO Virtualization: SR-IOV
Packet drops are seen for every 20-30 seconds once. Around 2K packets are
getting dropped in each instance. Please let me know if anybody has seen
this kind of issue in past and any pointers to debug this is really useful.
This is Gen2.0 x8 PCI. Attached the lspci -vvv output for reference.
Thanks for the help,
Swamy SRK
On Fri, May 20, 2016 at 2:47 PM, SwamZ <swamssrk@gmail.com> wrote:
> Hi Alex and Bruce,
> Thanks for the reply.
>
> Alex,
> Just saw your email. You are right, this issue was happening for
> multicast packets only. On 82599 NIC, multicast and broadcast packet
> replication (IXGBE_VT_CTL_REPLEN) was enabled even for single VF case,
> hence NIC was dropping the packets as rx_missed_errors. After disabling the
> replication, we are not seeing any drops.
>
> Regards,
> Swamy SRK
>
> On Fri, May 20, 2016 at 9:45 AM, Alexander Duyck <
> alexander.duyck@gmail.com> wrote:
>
>> On Fri, May 20, 2016 at 2:09 AM, SwamZ <swamssrk@gmail.com> wrote:
>> > Hi,
>> >
>> >
>> > While doing performance testing with larger packet size (like 4000
>> bytes),
>> > we are seeing rx_missed_errors on the interface. This issue is not seen
>> > with packet size less than 2000. There were questions asked in this
>> forum
>> > on rx_missed_error, but there was not any conclusion. Please let me know
>> > what could be the reason for these drops.
>> >
>> >
>> > I tried the following without any luck:
>> >
>> > 1) Tried different burst size like 16, 32 an 64
>> >
>> > 2) Tried different number of the rx descriptors like 512, 1024, 2048 and
>> > 4096.
>>
>> I've seen issues like this occur on the 82599 using the Linux kernel
>> diver in the past, but it was usually related to CPU C states. I'm
>> assuming that shouldn't apply since you are using the poll mode driver
>> right so there are no interrupts and the CPUs are busy polling.
>> Another thing to check is that you have sufficient PCIe bandwidth.
>> Take a look at an lspci -vvv for the device and make certain it is
>> connected at x8 Gen2.
>>
>> > Setup and testing details:
>> >
>> > CPU Speed : 2.6 GHz
>> >
>> > NIC : 82599ES 10-Gigabit
>> >
>> > IO Virtualization: SR-IOV
>> >
>> > IO and packet processing cores are in the same NUMA.
>>
>> If you are running SR-IOV then the SRRCTL_DROP_EN bit should be set in
>> all the SRRCTL registers if I am not mistaken. As such if you are
>> triggering Rx missed errors it may not be related to the driver
>> directly but could point to some sort of PCIe but utilization issue.
>> How many VFs are you using and is any of this traffic multicast or
>> broadcast? Multicast or broadcast traffic can potentially saturate
>> the PCIe bus if there is enough of it on the wire as each packet is
>> normally replicated for each active VF. So 1 packet on the wire (4K)
>> can become 8 on the PCIe bus (32K) if you have 7 active VFs.
>>
>> > Packet size : 4000 bytes
>> >
>> > Traffic rate: 80 % line rate. If the traffic rate is < ~80% then drops
>> are
>> > not seen.
>> >
>> > Application: This is a sample application developed using L3fwd example
>> app.
>> >
>> > DPDK version: 1.7
>> > Thanks,
>> > Swamy
>>
>
>
>
> --
> Swamz
>
--
Swamz
[-- Attachment #2: lspci_vvv.txt --]
[-- Type: text/plain, Size: 4863 bytes --]
88:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
Network Connection (rev 01)
Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
Physical Slot: 4
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 17
Region 0: Memory at c8100000 (32-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at a020 [size=32]
Region 3: Memory at c8204000 (32-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at c8280000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s
unlimited, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt-
ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not
Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF
Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance-
ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-,
EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Vital Product Data
Product Name: HP Ethernet 10Gb 2-port 560SFP+ Adapter
Read-only fields:
[PN] Part number: 665247-001
[EC] Engineering changes: D-5514
[SN] Serial number: MYI52608QY
[V0] Vendor specific: 11W/8W PCIeG2x8 2p 10Gb SFP+ Intel 82599
[V2] Vendor specific: 5526
[V4] Vendor specific: 00110A6786D8
[V5] Vendor specific: 0D
[RV] Reserved: checksum good, 0 byte(s) reserved
Read/write fields:
[V1] Vendor specific: 4.5.19
[V3] Vendor specific: 3.0.24
[V6] Vendor specific: 2.3.20
[YA] Asset tag: N/A
[YB] System specific: xxxxxxxxxxxxxxxx
[YC] System specific: xxxxxxxxxxxxxxxx
End
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+
MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 00-11-0a-ff-ff-67-86-d8
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 4, Function Dependency
Link: 00
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 000003bfffe00000 (64-bit, prefetchable)
Region 3: Memory at 000003bfffd00000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-05-31 18:50 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-05-20 9:09 Rx_missed_errors drops with larger packet size SwamZ
2016-05-20 10:44 ` Bruce Richardson
2016-05-20 16:45 ` Alexander Duyck
2016-05-20 21:47 ` SwamZ
2016-05-31 18:50 ` SwamZ
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.