Netdev Archive on lore.kernel.org
 help / color / Atom feed
From: Eric Dumazet <edumazet@google.com>
To: Heiner Kallweit <hkallweit1@gmail.com>
Cc: Charles DAYMAND <charles.daymand@wifirst.fr>,
	netdev <netdev@vger.kernel.org>
Subject: Re: [PATCH net] r8169: fix multicast tx issue with macvlan interface
Date: Mon, 13 Apr 2020 11:20:43 -0700
Message-ID: <CANn89iKA8MAef-XfkbLG3W+3=qUx4pqmKuWPBfrxAcupohLkyA@mail.gmail.com> (raw)
In-Reply-To: <297e210f-1784-44a9-17fb-7fbe8b6f9ec3@gmail.com>

On Mon, Apr 13, 2020 at 11:06 AM Heiner Kallweit <hkallweit1@gmail.com> wrote:
>
> On 06.04.2020 16:12, Charles DAYMAND wrote:
> > Hello,
> > Sorry for the long delay, I didn't have physical access to the
> > hardware last week. I did more testing today by connecting directly my
> > laptop to the hardware and the issue not only affects multicast but
> > also unicast with macvlan.
> > Only ping packets are correctly sent.
> > Using wireshark I can only see malformed ethernet packets (example
> > with multicast packet below)
> > Frame 1: 106 bytes on wire (848 bits), 106 bytes captured (848 bits)
> > IEEE 802.3 Ethernet
> >     Destination: IPv4mcast_09 (01:00:5e:00:00:09)
> >     Source: b2:41:6f:04:c1:86 (b2:41:6f:04:c1:86)
> >     Length: 96
> >         [Expert Info (Error/Malformed): Length field value goes past
> > the end of the payload]
> > Logical-Link Control
> >     DSAP: Unknown (0x45)
> >     SSAP: Unknown (0xc0)
> >     Control field: I, N(R)=46, N(S)=0 (0x5C00)
> > Data (88 bytes)
> >     Data: 50320c780111087fac159401e0000009020802080048207a...
> >     [Length: 88]
> > Please find the full pcap file at this url with also a tentative to
> > establish a ssh connection :
> > https://fourcot.fr/temp/malformed_ethernet2.pcap
> >
>
> A problem with HW checksumming should manifest as checksum failure,
> I don't think HW checksumming touches length information in headers.
> You could enable tracing network events to see whether packets look
> suspicious before reaching the network driver.
> (or do some printf debugging for the affected multicast packets)
>
> Because you initially suspected a problem with HW checksumming
> in a specific RTL8168 chip variant:
> - Check macvlan with another network driver supporting HW checksumming
> - Check macvlan with r8169 and another RTL8168 chip variant
>
> The fact that the issue doesn't exist w/o macvlan seems to indicate
> that the network driver can't be blamed here.

trace includes evidence of 802.1Q traffic though, maybe "macvlan" is a
red herring,
some confusion on wording maybe...

802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 2833
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 1011 0001 0001 = ID: 2833


>
> > Best regards
> >
> > Charles
> >
> >
> > Le mar. 31 mars 2020 à 16:07, Heiner Kallweit <hkallweit1@gmail.com> a écrit :
> >>
> >> Thanks for further testing! The good news from my perspective is that the issue doesn't occur
> >> w/o macvlen, therefore it doesn't seem to be a r8169 network driver issue.
> >>
> >> W/o knowing tcpdump in detail I think it switches the NIC to promiscuous mode, means
> >> it should see all packets, incl. the ones with checksum errors.
> >> Maybe you can mirror the port to which the problematic system is connected and
> >> analyze the traffic. Or for whatever reason the switch doesn't forward the multicast
> >> packets to your notebook.
> >>
> >> Heiner
> >>
> >>
> >> On 31.03.2020 15:44, Charles DAYMAND wrote:
> >>> Hello,
> >>> We tested to enable tx checksumming manually (via ethtool) on a kernel 4.19.0-5-amd64 which is the oldest kernel compatible with our software and we observed exactly the same issue.
> >>> For information when connecting a laptop directly to the interface we can't see any multicast packet when tx checksumming is enabled on tcpdump.
> >>> Our network is composed of a cisco switch and we can still see the multicast counters correctly increasing even when we have the issue.
> >>>
> >>> I also confirm that when not using macvlan but the real interface there is no issue on the multicast packets, they are correctly sent and received.
> >>> I have a stupid question, if the IP checksum was bad on the multicast packet, would the receiver NIC drop the packet or would it be seen by tcpdump by the receiver ?
> >>>
> >>> Le ven. 27 mars 2020 à 20:43, Eric Dumazet <edumazet@google.com <mailto:edumazet@google.com>> a écrit :
> >>>
> >>>     On Fri, Mar 27, 2020 at 12:17 PM Heiner Kallweit <hkallweit1@gmail.com <mailto:hkallweit1@gmail.com>> wrote:
> >>>     >
> >>>     > On 27.03.2020 19:52, Eric Dumazet wrote:
> >>>     > > On Fri, Mar 27, 2020 at 10:41 AM Heiner Kallweit <hkallweit1@gmail.com <mailto:hkallweit1@gmail.com>> wrote:
> >>>     > >>
> >>>     > >> On 27.03.2020 10:39, Heiner Kallweit wrote:
> >>>     > >>> On 27.03.2020 10:08, Charles Daymand wrote:
> >>>     > >>>> During kernel upgrade testing on our hardware, we found that macvlan
> >>>     > >>>> interface were no longer able to send valid multicast packet.
> >>>     > >>>>
> >>>     > >>>> tcpdump run on our hardware was correctly showing our multicast
> >>>     > >>>> packet but when connecting a laptop to our hardware we didn't see any
> >>>     > >>>> packets.
> >>>     > >>>>
> >>>     > >>>> Bisecting turned up commit 93681cd7d94f
> >>>     > >>>> "r8169: enable HW csum and TSO" activates the feature NETIF_F_IP_CSUM
> >>>     > >>>> which is responsible for the drop of packet in case of macvlan
> >>>     > >>>> interface. Note that revision RTL_GIGA_MAC_VER_34 was already a specific
> >>>     > >>>> case since TSO was keep disabled.
> >>>     > >>>>
> >>>     > >>>> Deactivating NETIF_F_IP_CSUM using ethtool is correcting our multicast
> >>>     > >>>> issue, but we believe that this hardware issue is important enough to
> >>>     > >>>> keep tx checksum off by default on this revision.
> >>>     > >>>>
> >>>     > >>>> The change is deactivating the default value of NETIF_F_IP_CSUM for this
> >>>     > >>>> specific revision.
> >>>     > >>>>
> >>>     > >>>
> >>>     > >>> The referenced commit may not be the root cause but just reveal another
> >>>     > >>> issue that has been existing before. Root cause may be in the net core
> >>>     > >>> or somewhere else. Did you check with other RTL8168 versions to verify
> >>>     > >>> that it's indeed a HW issue with this specific chip version?
> >>>     > >>>
> >>>     > >>> What you could do: Enable tx checksumming manually (via ethtool) on
> >>>     > >>> older kernel versions and check whether they are fine or not.
> >>>     > >>> If an older version is fine, then you can start a new bisect with tx
> >>>     > >>> checksumming enabled.
> >>>     > >>>
> >>>     > >>> And did you capture and analyze traffic to verify that actually the
> >>>     > >>> checksum is incorrect (and packets discarded therefore on receiving end)?
> >>>     > >>>
> >>>     > >>>
> >>>     > >>>> Fixes: 93681cd7d94f ("r8169: enable HW csum and TSO")
> >>>     > >>>> Signed-off-by: Charles Daymand <charles.daymand@wifirst.fr <mailto:charles.daymand@wifirst.fr>>
> >>>     > >>>> ---
> >>>     > >>>>  net/drivers/net/ethernet/realtek/r8169_main.c | 3 +++
> >>>     > >>>>  1 file changed, 3 insertions(+)
> >>>     > >>>>
> >>>     > >>>> diff --git a/net/drivers/net/ethernet/realtek/r8169_main.c b/net/drivers/net/ethernet/realtek/r8169_main.c
> >>>     > >>>> index a9bdafd15a35..3b69135fc500 100644
> >>>     > >>>> --- a/net/drivers/net/ethernet/realtek/r8169_main.c
> >>>     > >>>> +++ b/net/drivers/net/ethernet/realtek/r8169_main.c
> >>>     > >>>> @@ -5591,6 +5591,9 @@ static int rtl_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
> >>>     > >>>>              dev->vlan_features &= ~(NETIF_F_ALL_TSO | NETIF_F_SG);
> >>>     > >>>>              dev->hw_features &= ~(NETIF_F_ALL_TSO | NETIF_F_SG);
> >>>     > >>>>              dev->features &= ~(NETIF_F_ALL_TSO | NETIF_F_SG);
> >>>     > >>>> +            if (tp->mac_version == RTL_GIGA_MAC_VER_34) {
> >>>     > >>>> +                    dev->features &= ~NETIF_F_IP_CSUM;
> >>>     > >>>> +            }
> >>>     > >>>>      }
> >>>     > >>>>
> >>>     > >>>>      dev->hw_features |= NETIF_F_RXALL;
> >>>     > >>>>
> >>>     > >>>
> >>>     > >>
> >>>     > >> After looking a little bit at the macvlen code I think there might be an
> >>>     > >> issue in it, but I'm not sure, therefore let me add Eric (as macvlen doesn't
> >>>     > >> seem to have a dedicated maintainer).
> >>>     > >>
> >>>     > >> r8169 implements a ndo_features_check callback that disables tx checksumming
> >>>     > >> for the chip version in question and small packets (due to a HW issue).
> >>>     > >> macvlen uses passthru_features_check() as ndo_features_check callback, this
> >>>     > >> seems to indicate to me that the ndo_features_check callback of lowerdev is
> >>>     > >> ignored. This could explain the issue you see.
> >>>     > >>
> >>>     > >
> >>>     > > macvlan_queue_xmit() calls dev_queue_xmit_accel() after switching skb->dev,
> >>>     > > so the second __dev_queue_xmit() should eventually call the real_dev
> >>>     > > ndo_features_check()
> >>>     > >
> >>>     > Thanks, Eric. There's a second path in macvlan_queue_xmit() calling
> >>>     > dev_forward_skb(vlan->lowerdev, skb). Does what you said apply also there?
> >>>
> >>>     This path wont send packets to the physical NIC, packets are injected
> >>>     back via dev_forward_skb()
> >>>
> >>>     >
> >>>     > Still I find it strange that a tx hw checksumming issue should affect multicasts
> >>>     > only. Also the chip version in question is quite common and I would expect
> >>>     > others to have hit the same issue.
> >>>     > Maybe best would be to re-test on the affected system w/o involving macvlen.
> >>>     >
> >>>     > >
> >>>     > >
> >>>     > >> Would be interesting to see whether it fixes your issue if you let the
> >>>     > >> macvlen ndo_features_check call lowerdev's ndo_features_check. Can you try this?
> >>>     > >>
> >>>     > >> By the way:
> >>>     > >> Also the ndo_fix_features callback of lowerdev seems to be ignored.
> >>>     >
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> logo wifirst <http://www.wifirst.fr/en>
> >>>
> >>> Charles Daymand
> >>>
> >>> Développeur infrastructure
> >>>
> >>> 26 rue de Berri 75008 Paris
> >>>
> >>> Assistance dédiée responsable de site - 01 70 70 46 70
> >>> Assistance utilisateur - 01 70 70 46 26
> >>>
> >>
> >
> >
>

  reply index

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-27  9:08 Charles Daymand
2020-03-27  9:39 ` Heiner Kallweit
2020-03-27 17:41   ` Heiner Kallweit
2020-03-27 18:52     ` Eric Dumazet
2020-03-27 19:17       ` Heiner Kallweit
2020-03-27 19:42         ` Eric Dumazet
     [not found]           ` <CAFJtzm03QpjGRs70tth26BdUFN_o8zsJOccbnA58ma+2uwiGcg@mail.gmail.com>
2020-03-31 13:48             ` Charles DAYMAND
2020-03-31 14:07             ` Heiner Kallweit
2020-04-06 14:12               ` Charles DAYMAND
2020-04-06 22:16                 ` Heiner Kallweit
2020-04-06 23:20                   ` Eric Dumazet
2020-04-07  6:22                     ` Heiner Kallweit
2020-04-07 22:40                       ` Heiner Kallweit
2020-04-10  9:53                         ` Charles DAYMAND
2020-04-13 18:06                 ` Heiner Kallweit
2020-04-13 18:20                   ` Eric Dumazet [this message]
2020-04-15 11:55                     ` Charles Daymand
2020-05-12 18:37                       ` Heiner Kallweit
2020-04-27 21:12                     ` Florent Fourcot
2020-04-30 16:50                       ` Heiner Kallweit

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CANn89iKA8MAef-XfkbLG3W+3=qUx4pqmKuWPBfrxAcupohLkyA@mail.gmail.com' \
    --to=edumazet@google.com \
    --cc=charles.daymand@wifirst.fr \
    --cc=hkallweit1@gmail.com \
    --cc=netdev@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Netdev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/netdev/0 netdev/git/0.git
	git clone --mirror https://lore.kernel.org/netdev/1 netdev/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 netdev netdev/ https://lore.kernel.org/netdev \
		netdev@vger.kernel.org
	public-inbox-index netdev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.netdev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git