Inconsistency in packet drop due to MTU (eth vs veth)

Inconsistency in packet drop due to MTU (eth vs veth)

[Fredrik Markstrom]

Hello,

I've noticed an inconsistency between how physical ethernet and veth handles mtu.

If I setup two physical interfaces (directly connected) with different mtu:s, only the size of the outgoing packets are limited by the mtu. But with veth a packet is dropped if the mtu of the receiving interface is smaller then the packet size. 

This seems inconsistent to me, but maybe there is a reason for it ? 

Can someone confirm if it's a deliberate inconsistency or just a side effect of using dev_forward_skb() ?

Example:

Using physical interfaces
=================
client> ip link set dev eth0 mtu 500
server> ip link set dev eth2 mtu 1500
server> ping -qc 2 -s 600 client
PING 135.15.35.74 (135.15.35.74) 1100(1128) bytes of data.
--- 135.15.35.74 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.818/0.864/0.910/0.046 ms

Using veth
=============
server> ip netns add clientns
server> ip link add name serverif  type veth peer name clientif
server> ip addr add 10.0.0.1/24 dev serverif
server> ip link set serverif mtu 1500 up
server> ip link set clientif up netns clientns
server> ip netns exec clientns ip link set clientif mtu 500
server> ip netns exec clientns ip addr add 10.0.0.74/24 dev clientif
server> ping -qc 2 -s 600 10.0.0.74
PING 10.0.0.74 (10.0.0.74) 600(628) bytes of data.
--- 10.0.0.74 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1008ms

/F

Re: Inconsistency in packet drop due to MTU (eth vs veth)

[Eric Dumazet]

On Thu, 2017-01-19 at 17:41 +0100, Fredrik Markstrom wrote:
> Hello,
> 
> I've noticed an inconsistency between how physical ethernet and veth handles mtu.
> 
> If I setup two physical interfaces (directly connected) with different mtu:s, only the size of the outgoing packets are limited by the mtu. But with veth a packet is dropped if the mtu of the receiving interface is smaller then the packet size. 
> 
> This seems inconsistent to me, but maybe there is a reason for it ? 
> 
> Can someone confirm if it's a deliberate inconsistency or just a side effect of using dev_forward_skb() ?

It looks this was added in commit
38d408152a86598a50680a82fe3353b506630409
("veth: Allow setting the L3 MTU")

But what was really needed here was a way to change MRU :(

Re: Inconsistency in packet drop due to MTU (eth vs veth)

[Fredrik Markstrom]

 ---- On Thu, 19 Jan 2017 19:53:47 +0100 Eric Dumazet <eric.dumazet@gmail.com> wrote ---- 
 > On Thu, 2017-01-19 at 17:41 +0100, Fredrik Markstrom wrote: 
 > > Hello, 
 > >  
 > > I've noticed an inconsistency between how physical ethernet and veth handles mtu. 
 > >  
 > > If I setup two physical interfaces (directly connected) with different mtu:s, only the size of the outgoing packets are limited by the mtu. But with veth a packet is dropped if the mtu of the receiving interface is smaller then the packet size.  
 > >  
 > > This seems inconsistent to me, but maybe there is a reason for it ?  
 > >  
 > > Can someone confirm if it's a deliberate inconsistency or just a side effect of using dev_forward_skb() ? 
 >  
 > It looks this was added in commit 
 > 38d408152a86598a50680a82fe3353b506630409 
 > ("veth: Allow setting the L3 MTU") 
 >  
 > But what was really needed here was a way to change MRU :( 

Ok, do we consider this correct and/or something we need to be backwards compatible with ? Is it insane to believe that we can fix this "inconsistency" by removing the check ?

The commit message reads "For consistency I drop packets on the receive side when they are larger than the MTU", do we know what it's supposed
to be consistent with or is that lost in history ?

/Fredrik

 >  
 >  
 >  
 > 

Re: Inconsistency in packet drop due to MTU (eth vs veth)

[Eric Dumazet]

On Tue, 2017-01-31 at 14:32 +0100, Fredrik Markstrom wrote:
>  ---- On Thu, 19 Jan 2017 19:53:47 +0100 Eric Dumazet <eric.dumazet@gmail.com> wrote ---- 
>  > On Thu, 2017-01-19 at 17:41 +0100, Fredrik Markstrom wrote: 
>  > > Hello, 
>  > >  
>  > > I've noticed an inconsistency between how physical ethernet and
> veth handles mtu. 
>  > >  
>  > > If I setup two physical interfaces (directly connected) with
> different mtu:s, only the size of the outgoing packets are limited by
> the mtu. But with veth a packet is dropped if the mtu of the receiving
> interface is smaller then the packet size.  
>  > >  
>  > > This seems inconsistent to me, but maybe there is a reason for
> it ?  
>  > >  
>  > > Can someone confirm if it's a deliberate inconsistency or just a
> side effect of using dev_forward_skb() ? 
>  >  
>  > It looks this was added in commit 
>  > 38d408152a86598a50680a82fe3353b506630409 
>  > ("veth: Allow setting the L3 MTU") 
>  >  
>  > But what was really needed here was a way to change MRU :( 
> 
> Ok, do we consider this correct and/or something we need to be
> backwards compatible with ? Is it insane to believe that we can fix
> this "inconsistency" by removing the check ?
> 
> The commit message reads "For consistency I drop packets on the
> receive side when they are larger than the MTU", do we know what it's
> supposed
> to be consistent with or is that lost in history ?

There is no consistency among existing Ethernet drivers.

Many ethernet drivers size the buffers they post in RX ring buffer
according to MTU.

If MTU is set to 1500, RX buffers are sized to be about 1536 bytes,
so you wont be able to receive a 1700 bytes frame.

I guess that you could add a specific veth attribute to precisely
control MRU, that would not break existing applications.

Re: Inconsistency in packet drop due to MTU (eth vs veth)

[Fredrik Markstrom]

/F


 ---- On Tue, 31 Jan 2017 17:27:09 +0100 Eric Dumazet <eric.dumazet@gmail.com> wrote ---- 
 > On Tue, 2017-01-31 at 14:32 +0100, Fredrik Markstrom wrote: 
 > >  ---- On Thu, 19 Jan 2017 19:53:47 +0100 Eric Dumazet <eric.dumazet@gmail.com> wrote ----  
 > >  > On Thu, 2017-01-19 at 17:41 +0100, Fredrik Markstrom wrote:  
 > >  > > Hello,  
 > >  > >   
 > >  > > I've noticed an inconsistency between how physical ethernet and 
 > > veth handles mtu.  
 > >  > >   
 > >  > > If I setup two physical interfaces (directly connected) with 
 > > different mtu:s, only the size of the outgoing packets are limited by 
 > > the mtu. But with veth a packet is dropped if the mtu of the receiving 
 > > interface is smaller then the packet size.   
 > >  > >   
 > >  > > This seems inconsistent to me, but maybe there is a reason for 
 > > it ?   
 > >  > >   
 > >  > > Can someone confirm if it's a deliberate inconsistency or just a 
 > > side effect of using dev_forward_skb() ?  
 > >  >   
 > >  > It looks this was added in commit  
 > >  > 38d408152a86598a50680a82fe3353b506630409  
 > >  > ("veth: Allow setting the L3 MTU")  
 > >  >   
 > >  > But what was really needed here was a way to change MRU :(  
 > >  
 > > Ok, do we consider this correct and/or something we need to be 
 > > backwards compatible with ? Is it insane to believe that we can fix 
 > > this "inconsistency" by removing the check ? 
 > >  
 > > The commit message reads "For consistency I drop packets on the 
 > > receive side when they are larger than the MTU", do we know what it's 
 > > supposed 
 > > to be consistent with or is that lost in history ? 
 >  
 > There is no consistency among existing Ethernet drivers. 
 >  
 > Many ethernet drivers size the buffers they post in RX ring buffer 
 > according to MTU. 
 >  
 > If MTU is set to 1500, RX buffers are sized to be about 1536 bytes, 
 > so you wont be able to receive a 1700 bytes frame. 
 >  
 > I guess that you could add a specific veth attribute to precisely 
 > control MRU, that would not break existing applications. 

Ok, I will propose a patch shortly. And thanks, your response time is
awesome !

/Fredrik

 >  
 >  
 >  
 > 

Re: Inconsistency in packet drop due to MTU (eth vs veth)

[Toshiaki Makita]

On 17/02/03 (金) 17:07, Fredrik Markstrom wrote:
>  ---- On Tue, 31 Jan 2017 17:27:09 +0100 Eric Dumazet <eric.dumazet@gmail.com> wrote ----
>  > On Tue, 2017-01-31 at 14:32 +0100, Fredrik Markstrom wrote:
>  > >  ---- On Thu, 19 Jan 2017 19:53:47 +0100 Eric Dumazet <eric.dumazet@gmail.com> wrote ----
>  > >  > On Thu, 2017-01-19 at 17:41 +0100, Fredrik Markstrom wrote:
>  > >  > > Hello,
>  > >  > >
>  > >  > > I've noticed an inconsistency between how physical ethernet and
>  > > veth handles mtu.
>  > >  > >
>  > >  > > If I setup two physical interfaces (directly connected) with
>  > > different mtu:s, only the size of the outgoing packets are limited by
>  > > the mtu. But with veth a packet is dropped if the mtu of the receiving
>  > > interface is smaller then the packet size.
>  > >  > >
>  > >  > > This seems inconsistent to me, but maybe there is a reason for
>  > > it ?
>  > >  > >
>  > >  > > Can someone confirm if it's a deliberate inconsistency or just a
>  > > side effect of using dev_forward_skb() ?
>  > >  >
>  > >  > It looks this was added in commit
>  > >  > 38d408152a86598a50680a82fe3353b506630409
>  > >  > ("veth: Allow setting the L3 MTU")
>  > >  >
>  > >  > But what was really needed here was a way to change MRU :(
>  > >
>  > > Ok, do we consider this correct and/or something we need to be
>  > > backwards compatible with ? Is it insane to believe that we can fix
>  > > this "inconsistency" by removing the check ?
>  > >
>  > > The commit message reads "For consistency I drop packets on the
>  > > receive side when they are larger than the MTU", do we know what it's
>  > > supposed
>  > > to be consistent with or is that lost in history ?
>  >
>  > There is no consistency among existing Ethernet drivers.
>  >
>  > Many ethernet drivers size the buffers they post in RX ring buffer
>  > according to MTU.
>  >
>  > If MTU is set to 1500, RX buffers are sized to be about 1536 bytes,
>  > so you wont be able to receive a 1700 bytes frame.
>  >
>  > I guess that you could add a specific veth attribute to precisely
>  > control MRU, that would not break existing applications.
>
> Ok, I will propose a patch shortly. And thanks, your response time is
> awesome !

But why do you want to configure MRU?
What is the problem with setting MTU instead.

Toshiaki Makita