From: Paolo Abeni <pabeni@redhat.com>
To: Matthias Treydte <mt@waldheinz.de>
Cc: stable@vger.kernel.org, netdev@vger.kernel.org,
regressions@lists.linux.dev, davem@davemloft.net,
yoshfuji@linux-ipv6.org, dsahern@kernel.org,
Willem de Bruijn <willemdebruijn.kernel@gmail.com>,
David Ahern <dsahern@gmail.com>
Subject: Re: [regression] UDP recv data corruption
Date: Fri, 02 Jul 2021 13:42:05 +0200 [thread overview]
Message-ID: <39df657bee89e56c704782e0c061383d276d2f7c.camel@redhat.com> (raw)
In-Reply-To: <CA+FuTSc3POcZo0En3JBqRwq2+eF645_Cs4U-4nBmTs9FvjoVkg@mail.gmail.com>
Hello,
On Thu, 2021-07-01 at 20:31 -0400, Willem de Bruijn wrote:
> On Thu, Jul 1, 2021 at 11:39 AM David Ahern <dsahern@gmail.com> wrote:
> > [ adding Paolo, author of 18f25dc39990 ]
> >
> > On 7/1/21 4:47 AM, Matthias Treydte wrote:
> > > Hello,
> > >
> > > we recently upgraded the Linux kernel from 5.11.21 to 5.12.12 in our
> > > video stream receiver appliance and noticed compression artifacts on
> > > video streams that were previously looking fine. We are receiving UDP
> > > multicast MPEG TS streams through an FFMpeg / libav layer which does the
> > > socket and lower level protocol handling. For affected kernels it spills
> > > the log with messages like
> > >
> > > > [mpegts @ 0x7fa130000900] Packet corrupt (stream = 0, dts = 6870802195).
> > > > [mpegts @ 0x7fa11c000900] Packet corrupt (stream = 0, dts = 6870821068).
> > >
> > > Bisecting identified commit 18f25dc399901426dff61e676ba603ff52c666f7 as
> > > the one introducing the problem in the mainline kernel. It was
> > > backported to the 5.12 series in
> > > 450687386cd16d081b58cd7a342acff370a96078. Some random observations which
> > > may help to understand what's going on:
> > >
> > > * the problem exists in Linux 5.13
> > > * reverting that commit on top of 5.13 makes the problem go away
> > > * Linux 5.10.45 is fine
> > > * no relevant output in dmesg
> > > * can be reproduced on different hardware (Intel, AMD, different
> > > NICs, ...)
> > > * we do use the bonding driver on the systems (but I did not yet
> > > verify that this is related)
> > > * we do not use vxlan (mentioned in the commit message)
> > > * the relevant code in FFMpeg identifying packet corruption is here:
> > >
> > > https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/mpegts.c#L2758
> > >
> > > And the bonding configuration:
> > >
> > > # cat /proc/net/bonding/bond0
> > > Ethernet Channel Bonding Driver: v5.10.45
> > >
> > > Bonding Mode: fault-tolerance (active-backup)
> > > Primary Slave: None
> > > Currently Active Slave: enp2s0
> > > MII Status: up
> > > MII Polling Interval (ms): 100
> > > Up Delay (ms): 0
> > > Down Delay (ms): 0
> > > Peer Notification Delay (ms): 0
> > >
> > > Slave Interface: enp2s0
> > > MII Status: up
> > > Speed: 1000 Mbps
> > > Duplex: full
> > > Link Failure Count: 0
> > > Permanent HW addr: 80:ee:73:XX:XX:XX
> > > Slave queue ID: 0
> > >
> > > Slave Interface: enp3s0
> > > MII Status: down
> > > Speed: Unknown
> > > Duplex: Unknown
> > > Link Failure Count: 0
> > > Permanent HW addr: 80:ee:73:XX:XX:XX
> > > Slave queue ID: 0
> > >
> > >
> > > If there is anything else I can do to help tracking this down please let
> > > me know.
Thank you for the report!
According to the above I would wild guess that the GRO layer is
wrongly/unexpectly aggregating some UDP packets, but I do agree with
Willem, it looks like the code pointed by the bisecting should not do
anything evil here, but I'm likely missing some items.
Could you please:
- tell how frequent is the pkt corruption, even a rough estimate of the
frequency.
- provide the features exposed by the relevant devices:
ethtool -k <nic name>
- check, if possibly, how exactly the pkts are corrupted. Wrong size?
bad csum? what else?
- ideally a short pcap trace comprising the problematic packets would
be great!
Thanks!
Paolo
next prev parent reply other threads:[~2021-07-02 11:42 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-01 10:47 [regression] UDP recv data corruption Matthias Treydte
2021-07-01 15:39 ` David Ahern
2021-07-02 0:31 ` Willem de Bruijn
2021-07-02 11:42 ` Paolo Abeni [this message]
2021-07-02 14:31 ` Matthias Treydte
2021-07-02 12:36 ` Matthias Treydte
2021-07-02 14:06 ` Paolo Abeni
2021-07-02 14:21 ` Paolo Abeni
2021-07-02 15:23 ` Matthias Treydte
2021-07-02 15:32 ` Paolo Abeni
2021-07-02 16:07 ` Matthias Treydte
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=39df657bee89e56c704782e0c061383d276d2f7c.camel@redhat.com \
--to=pabeni@redhat.com \
--cc=davem@davemloft.net \
--cc=dsahern@gmail.com \
--cc=dsahern@kernel.org \
--cc=mt@waldheinz.de \
--cc=netdev@vger.kernel.org \
--cc=regressions@lists.linux.dev \
--cc=stable@vger.kernel.org \
--cc=willemdebruijn.kernel@gmail.com \
--cc=yoshfuji@linux-ipv6.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).