EoGRE sends undersized frames without padding

EoGRE sends undersized frames without padding

[Steinar H. Gunderson]

Hi,

I'm trying to connect some VMs over EoGRE (using gretap on my side):

  ip link add foo type gretap remote <remote> local <local>

This works fine for large packets, but the system in the other end
drops smaller packets, such as ARP requests and small ICMP pings.

After looking at the GRE packets in Wireshark, it turns out the Ethernet
packets within the EoGRE packet is undersized (under 60 bytes), and Linux
doesn't pad them. I haven't found anything in RFC 7637 that says anything
about padding, so I would assume it should conform to the usual Ethernet
padding rules, ie., pad to at least ETH_ZLEN. However, nothing in Linux' IP
stack seems to actually do this, which means that when the packet is
decapsulated in the other end and put on the (potentially virtual) wire,
it gets dropped. The other system properly pads its small frames when sending
them.

Is there a way to get around this, short of looping the packets out through a
physical wire to get the padding? Is it simply a bug? I've been testing with
4.19.28, but it doesn't look like git master has any changes in this area.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: EoGRE sends undersized frames without padding

[Steinar H. Gunderson]

On Thu, May 30, 2019 at 10:35:08AM +0200, Steinar H. Gunderson wrote:
> After looking at the GRE packets in Wireshark, it turns out the Ethernet
> packets within the EoGRE packet is undersized (under 60 bytes), and Linux
> doesn't pad them. I haven't found anything in RFC 7637 that says anything
> about padding, so I would assume it should conform to the usual Ethernet
> padding rules, ie., pad to at least ETH_ZLEN. However, nothing in Linux' IP
> stack seems to actually do this, which means that when the packet is
> decapsulated in the other end and put on the (potentially virtual) wire,
> it gets dropped. The other system properly pads its small frames when sending
> them.

As a proof of concept (no error handling, probably poor performance, not
implemented for IPv6, other issues?), this patch works and fixes my problem:

diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
index 4b0526441476..00be99d23e5c 100644
--- a/net/ipv4/ip_gre.c
+++ b/net/ipv4/ip_gre.c
@@ -441,6 +441,11 @@ static void __gre_xmit(struct sk_buff *skb, struct net_device *dev,
 	if (tunnel->parms.o_flags & TUNNEL_SEQ)
 		tunnel->o_seqno++;
 
+	if (proto == htons(ETH_P_TEB) && skb->len < ETH_ZLEN) {
+		skb_cow(skb, dev->needed_headroom + ETH_ZLEN - skb->len);
+		skb_put_zero(skb, ETH_ZLEN - skb->len);
+	}
+
 	/* Push GRE header. */
 	gre_build_header(skb, tunnel->tun_hlen,
 			 tunnel->parms.o_flags, proto, tunnel->parms.o_key,

If I apply the patch on the side with the gretap tunnel, the ARP packets are
properly padded, and seen by the host in the other end.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: EoGRE sends undersized frames without padding

[Steinar H. Gunderson]

On Fri, May 31, 2019 at 12:38:32AM +0200, Steinar H. Gunderson wrote:
> As a proof of concept (no error handling, probably poor performance, not
> implemented for IPv6, other issues?), this patch works and fixes my problem:

Hi,

Any comments on this?

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: EoGRE sends undersized frames without padding

[Cong Wang]

On Thu, May 30, 2019 at 2:08 AM Steinar H. Gunderson
<steinar+kernel@gunderson.no> wrote:
>
> Hi,
>
> I'm trying to connect some VMs over EoGRE (using gretap on my side):
>
>   ip link add foo type gretap remote <remote> local <local>
>
> This works fine for large packets, but the system in the other end
> drops smaller packets, such as ARP requests and small ICMP pings.

Is the other end Linux too?

>
> After looking at the GRE packets in Wireshark, it turns out the Ethernet
> packets within the EoGRE packet is undersized (under 60 bytes), and Linux
> doesn't pad them. I haven't found anything in RFC 7637 that says anything
> about padding, so I would assume it should conform to the usual Ethernet
> padding rules, ie., pad to at least ETH_ZLEN. However, nothing in Linux' IP
> stack seems to actually do this, which means that when the packet is
> decapsulated in the other end and put on the (potentially virtual) wire,
> it gets dropped. The other system properly pads its small frames when sending
> them.


If the packet doesn't go through any real wire, it could still be accepted
by Linux even when it is smaller than ETH_ZLEN, I think. Some hardware
switches pad for ETH_ZLEN when it goes through a real wire.

So, how is your packet routed between different VM? Via a Linux bridge?


>
> Is there a way to get around this, short of looping the packets out through a
> physical wire to get the padding? Is it simply a bug? I've been testing with
> 4.19.28, but it doesn't look like git master has any changes in this area.
>

It is still too early to say it is a bug. Is this a regression?

Thanks.

Re: EoGRE sends undersized frames without padding

[Steinar H. Gunderson]

On Tue, Jun 04, 2019 at 09:31:42PM -0700, Cong Wang wrote:
>> This works fine for large packets, but the system in the other end
>> drops smaller packets, such as ARP requests and small ICMP pings.
> Is the other end Linux too?

Yes and no. The other end is Linux with Open vSwitch, which sends the packet
on to a VM. The VM is an appliance which I do not control, and while the
management plane runs Linux, the data plane is as far as I know implemented
in userspace.

The appliance itself can also run EoGRE directly, but I haven't gotten it to
work.

> If the packet doesn't go through any real wire, it could still be accepted
> by Linux even when it is smaller than ETH_ZLEN, I think.

Yes, but that's just Linux accepting something invalid, no? It doesn't mean
it should be sending it out.

> Some hardware switches pad for ETH_ZLEN when it goes through a real wire.

All hardware switches should; it's a 802.1Q demand. (Some have traditionally
been buggy in that they haven't added extra padding back when they strip the
VLAN tag.)

> It is still too early to say it is a bug. Is this a regression?

I haven't tried it with earlier versions; I would suspect it's not a
regression.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: EoGRE sends undersized frames without padding

[Cong Wang]

On Wed, Jun 5, 2019 at 12:27 AM Steinar H. Gunderson
<steinar+kernel@gunderson.no> wrote:
>
> On Tue, Jun 04, 2019 at 09:31:42PM -0700, Cong Wang wrote:
> >> This works fine for large packets, but the system in the other end
> >> drops smaller packets, such as ARP requests and small ICMP pings.
> > Is the other end Linux too?
>
> Yes and no. The other end is Linux with Open vSwitch, which sends the packet
> on to a VM. The VM is an appliance which I do not control, and while the
> management plane runs Linux, the data plane is as far as I know implemented
> in userspace.

Hmm, sounds like openvswitch should pad the packets in this scenario,
like hardware switches padding those on real wires.


>
> The appliance itself can also run EoGRE directly, but I haven't gotten it to
> work.
>
> > If the packet doesn't go through any real wire, it could still be accepted
> > by Linux even when it is smaller than ETH_ZLEN, I think.
>
> Yes, but that's just Linux accepting something invalid, no? It doesn't mean
> it should be sending it out.

Well, we can always craft our own ill-formatted packets, right? :) Does
any standard say OS has to drop ethernet frames shorter than the
minimum?


>
> > Some hardware switches pad for ETH_ZLEN when it goes through a real wire.
>
> All hardware switches should; it's a 802.1Q demand. (Some have traditionally
> been buggy in that they haven't added extra padding back when they strip the
> VLAN tag.)

If so, so is the software switch, that is openvswitch?

>
> > It is still too early to say it is a bug. Is this a regression?
>
> I haven't tried it with earlier versions; I would suspect it's not a
> regression.

Good to know.

Thanks.

Re: EoGRE sends undersized frames without padding

[Steinar H. Gunderson]

On Wed, Jun 05, 2019 at 06:17:51PM -0700, Cong Wang wrote:
> Hmm, sounds like openvswitch should pad the packets in this scenario,
> like hardware switches padding those on real wires.

Well, openvswitch say that they just throw packets around and assume they're
valid... :-)

In any case, if you talk EoGRE to the vWLC directly, I doubt it accepts this,
given that it doesn't accept it on the virtual NICs.

>> Yes, but that's just Linux accepting something invalid, no? It doesn't mean
>> it should be sending it out.
> Well, we can always craft our own ill-formatted packets, right? :) Does
> any standard say OS has to drop ethernet frames shorter than the
> minimum?

I believe you're fully allowed to accept them (although it might be
technically difficult on physical media). But that doesn't mean everybody
else has to accept them. :-)

>>> Some hardware switches pad for ETH_ZLEN when it goes through a real wire.
>> All hardware switches should; it's a 802.1Q demand. (Some have traditionally
>> been buggy in that they haven't added extra padding back when they strip the
>> VLAN tag.)
> If so, so is the software switch, that is openvswitch?

What if the other end isn't a (virtual) switch, but a host?

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: EoGRE sends undersized frames without padding

[Cong Wang]

On Thu, Jun 6, 2019 at 12:36 AM Steinar H. Gunderson
<steinar+kernel@gunderson.no> wrote:
>
> On Wed, Jun 05, 2019 at 06:17:51PM -0700, Cong Wang wrote:
> > Hmm, sounds like openvswitch should pad the packets in this scenario,
> > like hardware switches padding those on real wires.
>
> Well, openvswitch say that they just throw packets around and assume they're
> valid... :-)

_If_ the hardware switch has to pad them (according to what you said),
why software switch doesn't?

>
> In any case, if you talk EoGRE to the vWLC directly, I doubt it accepts this,
> given that it doesn't accept it on the virtual NICs.
>
> >> Yes, but that's just Linux accepting something invalid, no? It doesn't mean
> >> it should be sending it out.
> > Well, we can always craft our own ill-formatted packets, right? :) Does
> > any standard say OS has to drop ethernet frames shorter than the
> > minimum?
>
> I believe you're fully allowed to accept them (although it might be
> technically difficult on physical media). But that doesn't mean everybody
> else has to accept them. :-)

Sure, Linux is already different with other OS'es, this also means Linux
doesn't have to reject them.

>
> >>> Some hardware switches pad for ETH_ZLEN when it goes through a real wire.
> >> All hardware switches should; it's a 802.1Q demand. (Some have traditionally
> >> been buggy in that they haven't added extra padding back when they strip the
> >> VLAN tag.)
> > If so, so is the software switch, that is openvswitch?
>
> What if the other end isn't a (virtual) switch, but a host?

Rather than arguing about this, please check what ethernet standard
says. It would be much easier to convince others with standard.

Depends on what standard says, we may need to pad on xmit path or on
forwarding path (switch), or rejecting shorter frames on receive path.

Thanks.

Re: EoGRE sends undersized frames without padding

[Steinar H. Gunderson]

On Fri, Jun 07, 2019 at 12:57:57PM -0700, Cong Wang wrote:
>> Well, openvswitch say that they just throw packets around and assume they're
>> valid... :-)
> _If_ the hardware switch has to pad them (according to what you said),
> why software switch doesn't?

Trust me, I'm telling them they have to deal with this, too. :-)

But you can't really assume there's a switch in the middle at all (whether
hardware nor software). You can connect a NIC to a NIC with no switch in-between.

Normally, it's not the switch that adds the padding; it's done in the NIC
(MAC sublayer), so if the switch doesn't modify the packet, it can just send
it on. However, there are situations where it has to, like for instance when
stripping a VLAN tag (802.1Q-2005, section 6.5.1), since that makes the
packet shorter.

> Rather than arguing about this, please check what ethernet standard
> says. It would be much easier to convince others with standard.

The Ethernet standard? That's pretty clear; Ethernet frames are a minimum of
64 bytes (including FCS at the end, so 60 payload bytes including MAC
addresses etc.). IEEE 802.3-2015 4.2.3.3:

  The CSMA/CD Media Access mechanism requires that a minimum frame length of
  minFrameSize bits be transmitted. If frameSize is less than minFrameSize,
  then the CSMA/CD MAC sublayer shall append extra bits in units of octets
  (Pad), after the end of the MAC Client Data field but prior to calculating
  and appending the FCS (if not provided by the MAC client). The number of
  extra bits shall be sufficient to ensure that the frame, from the DA field
  through the FCS field inclusive, is at least minFrameSize bits.

4A.2.3.2.4 also says, without assuming anything about CSMA/CD:

  The MAC requires that a minimum frame length of minFrameSize bits be
  transmitted. If frameSize is less than minFrameSize, then the MAC sublayer
  shall append extra bits in units of octets (pad), after the end of the MAC
  client data field but prior to calculating, and appending, the FCS (if not
  provided by the MAC client). The number of extra bits shall be sufficient
  to ensure that the frame, from the DA field through the FCS field
  inclusive, is at least minFrameSize bits. If the FCS is (optionally)
  provided by the MAC client, the pad shall also be provided by the MAC
  client. The content of the pad is unspecified.

minFrameSize is defined in 4A.4.2:

  minFrameSize   512 bits (64 octets)

As for what to do on undersized packets, section 4.2.9 contains this
pseudocode:

  receiveSucceeding := (frameSize ≥ minFrameSize) {Reject frames too small}

To be honest, I don't see that dropping undersized frames gives any
real-world gains in the case of EoGRE, though. I'd say that the most
reasonable thing to do would be to pad on transmit, and accept undersized
frames on receive. You could argue that's wasteful for cases like the
loopback interface, but honestly, you never really know what people are going
to do with the packets (just consider the case of tcpreplay).

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: EoGRE sends undersized frames without padding

[Gregory Rose]

On 6/7/2019 12:57 PM, Cong Wang wrote:
> On Thu, Jun 6, 2019 at 12:36 AM Steinar H. Gunderson
> <steinar+kernel@gunderson.no> wrote:
>> On Wed, Jun 05, 2019 at 06:17:51PM -0700, Cong Wang wrote:
>>> Hmm, sounds like openvswitch should pad the packets in this scenario,
>>> like hardware switches padding those on real wires.
>> Well, openvswitch say that they just throw packets around and assume they're
>> valid... :-)
> _If_ the hardware switch has to pad them (according to what you said),
> why software switch doesn't?

Well one thing I can think of is that there are things that can be done 
in HW that become very
expensive in SW.  CRC checking and packet padding are expensive in SW.

>
>> In any case, if you talk EoGRE to the vWLC directly, I doubt it accepts this,
>> given that it doesn't accept it on the virtual NICs.
>>
>>>> Yes, but that's just Linux accepting something invalid, no? It doesn't mean
>>>> it should be sending it out.
>>> Well, we can always craft our own ill-formatted packets, right? :) Does
>>> any standard say OS has to drop ethernet frames shorter than the
>>> minimum?
>> I believe you're fully allowed to accept them (although it might be
>> technically difficult on physical media). But that doesn't mean everybody
>> else has to accept them. :-)
> Sure, Linux is already different with other OS'es, this also means Linux
> doesn't have to reject them.
>
>>>>> Some hardware switches pad for ETH_ZLEN when it goes through a real wire.
>>>> All hardware switches should; it's a 802.1Q demand. (Some have traditionally
>>>> been buggy in that they haven't added extra padding back when they strip the
>>>> VLAN tag.)
>>> If so, so is the software switch, that is openvswitch?
>> What if the other end isn't a (virtual) switch, but a host?
> Rather than arguing about this, please check what ethernet standard
> says. It would be much easier to convince others with standard.
>
> Depends on what standard says, we may need to pad on xmit path or on
> forwarding path (switch), or rejecting shorter frames on receive path.
>
> Thanks.

I am used to Ethernet switches dropping undersized and bad CRC frames.  
Sure, if the incoming frame is
valid and then because of transformations in the packet due to 802.1q 
specs or any other reasons (NSH comes
mind) then the packet must be padded and the correct checksum computed 
by the switch.  That's fine.

But if the incoming packet to the switch is malformed (undersized and 
bad CRC in this case) then switches can
and probably will drop the packet.

It appears that openvswitch does not.  Steinar and I are discussing that 
situation on a different list.

Thanks,

- Greg