* What would cause Ubuntu 18.04 to perform different fragmentation when forwarding to WireGuard tunnel, depending on packet source?
@ 2019-11-27 7:55 Sander Saares
0 siblings, 0 replies; only message in thread
From: Sander Saares @ 2019-11-27 7:55 UTC (permalink / raw)
To: wireguard
[-- Attachment #1.1: Type: text/plain, Size: 4548 bytes --]
Hi!
I am troubleshooting some odd behavior and noticed that when my Ubuntu 18.04 server is routing packets between the internet and a WireGuard tunnel, there exist some odd differences between how different connections are handled that I cannot account for.
In both cases, I have a client (192.168.147.14) from the other side of the tunnel making an HTTPS connection. The IPv4 packets received from the internet have the "don't fragment" flag set, as is common. What I find is:
* When processing large packets from server A (159.148.147.196), Ubuntu responds with "needs fragmentation" and correctly gives the WireGuard MTU of 1420 as the next hop MTU.
* When processing large packets from server B (40.68.232.16), Ubuntu just does the fragmentation itself and forwards the packets no questions asked. Despite "don't fragment" bit being set.
I am unable to explain the difference in behavior. What am I missing here? Why might the two cases be handled differently by the operating system? I do not even know if this is WireGuard related - maybe it also occurs with non-WireGuard adapters.
My forwarding-relevant WireGuard configuration is simply the following:
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
I have not adjusted any relevant operating system settings from defaults other than allowing IP forwarding.
The initial packets of the connection that do not exceed the WireGuard MTU are correctly routed through the WireGuard tunnel in both cases.
Response from server A
Internet Protocol Version 4, Src: 159.148.147.196, Dst: 172.16.21.250
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 3042
Identification: 0x2839 (10297)
Flags: 0x4000, Don't fragment
...0 0000 0000 0000 = Fragment offset: 0
Time to live: 46
Protocol: TCP (6)
Header checksum: 0x237a [validation disabled]
[Header checksum status: Unverified]
Source: 159.148.147.196
Destination: 172.16.21.250
In response, "fragmentation needed" is sent
Internet Protocol Version 4, Src: 172.16.21.250, Dst: 159.148.147.196
Internet Control Message Protocol
Type: 3 (Destination unreachable)
Code: 4 (Fragmentation needed)
Checksum: 0x99ba [correct]
[Checksum Status: Good]
Unused: a8
Length: 246
[Length of original datagram: 984]
MTU of next hop: 1420
Internet Protocol Version 4, Src: 159.148.147.196, Dst: 172.16.21.250
Transmission Control Protocol, Src Port: 443, Dst Port: 43254, Seq: 2041044714, Ack: 1154907605
Transport Layer Security
Response from server B
Internet Protocol Version 4, Src: 40.68.232.16, Dst: 172.16.21.250
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 5716
Identification: 0x22e1 (8929)
Flags: 0x4000, Don't fragment
...0 0000 0000 0000 = Fragment offset: 0
Time to live: 119
Protocol: TCP (6)
Header checksum: 0xf863 [validation disabled]
[Header checksum status: Unverified]
Source: 40.68.232.16
Destination: 172.16.21.250
In response, Ubuntu just fragments the packet anyway and pushes through the WG tunnel
22 0.081525663 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=11, datalen=1408
23 0.081532164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=12, datalen=1408
24 0.081535164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=13, datalen=1408
25 0.081538164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=14, datalen=1408
26 0.081541064 172.16.21.250 192.168.147.14 WireGuard 410 Transport Data, receiver=0xD8778C02, counter=15, datalen=336
Cheers,
Sander Saares
Advisor
Axinom | Soola 8 | 51004 Tartu | Estonia
phone: +49 911 80109-54
saares@axinom.com<mailto:saares@axinom.com> | www.axinom.com<http://www.axinom.com/>
Managing Directors: Sergei Gussev, Oleg Knut
Tartu Circuit Court, Reg. 11046287
[-- Attachment #1.2: Type: text/html, Size: 14940 bytes --]
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1170170703;
mso-list-type:hybrid;
mso-list-template-ids:-1522905622 1350069654 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:Arial;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am troubleshooting some odd behavior and noticed that when my Ubuntu 18.04 server is routing packets between the internet and a WireGuard tunnel, there exist some odd differences between how different connections are handled that I cannot
account for.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In both cases, I have a client (192.168.147.14) from the other side of the tunnel making an HTTPS connection. The IPv4 packets received from the internet have the “don’t fragment” flag set, as is common. What I find is:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">When processing large packets from server A (159.148.147.196), Ubuntu responds with “needs fragmentation” and correctly gives the WireGuard MTU of 1420 as the next hop MTU.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">When processing large packets from server B (40.68.232.16), Ubuntu just does the fragmentation itself and forwards the packets no questions asked. Despite “don’t fragment” bit being
set.<o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am unable to explain the difference in behavior. What am I missing here? Why might the two cases be handled differently by the operating system? I do not even know if this is WireGuard related – maybe it also occurs with non-WireGuard
adapters.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My forwarding-relevant WireGuard configuration is simply the following:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE<o:p></o:p></p>
<p class="MsoNormal">PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have not adjusted any relevant operating system settings from defaults other than allowing IP forwarding.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The initial packets of the connection that do not exceed the WireGuard MTU are correctly routed through the WireGuard tunnel in both cases.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Response from server A<o:p></o:p></b></p>
<p class="MsoNormal">Internet Protocol Version 4, Src: 159.148.147.196, Dst: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"> 0100 .... = Version: 4<o:p></o:p></p>
<p class="MsoNormal"> .... 0101 = Header Length: 20 bytes (5)<o:p></o:p></p>
<p class="MsoNormal"> Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)<o:p></o:p></p>
<p class="MsoNormal"> Total Length: 3042<o:p></o:p></p>
<p class="MsoNormal"> Identification: 0x2839 (10297)<o:p></o:p></p>
<p class="MsoNormal"><b> Flags: 0x4000, Don't fragment<o:p></o:p></b></p>
<p class="MsoNormal"> ...0 0000 0000 0000 = Fragment offset: 0<o:p></o:p></p>
<p class="MsoNormal"> Time to live: 46<o:p></o:p></p>
<p class="MsoNormal"> Protocol: TCP (6)<o:p></o:p></p>
<p class="MsoNormal"> Header checksum: 0x237a [validation disabled]<o:p></o:p></p>
<p class="MsoNormal"> [Header checksum status: Unverified]<o:p></o:p></p>
<p class="MsoNormal"> Source: 159.148.147.196<o:p></o:p></p>
<p class="MsoNormal"> Destination: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>In response, “fragmentation needed” is sent<o:p></o:p></b></p>
<p class="MsoNormal">Internet Protocol Version 4, Src: 172.16.21.250, Dst: 159.148.147.196<o:p></o:p></p>
<p class="MsoNormal">Internet Control Message Protocol<o:p></o:p></p>
<p class="MsoNormal"> Type: 3 (Destination unreachable)<o:p></o:p></p>
<p class="MsoNormal"> Code: 4 (Fragmentation needed)<o:p></o:p></p>
<p class="MsoNormal"> Checksum: 0x99ba [correct]<o:p></o:p></p>
<p class="MsoNormal"> [Checksum Status: Good]<o:p></o:p></p>
<p class="MsoNormal"> Unused: a8<o:p></o:p></p>
<p class="MsoNormal"> Length: 246<o:p></o:p></p>
<p class="MsoNormal"> [Length of original datagram: 984]<o:p></o:p></p>
<p class="MsoNormal"> MTU of next hop: 1420<o:p></o:p></p>
<p class="MsoNormal"> Internet Protocol Version 4, Src: 159.148.147.196, Dst: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"> Transmission Control Protocol, Src Port: 443, Dst Port: 43254, Seq: 2041044714, Ack: 1154907605<o:p></o:p></p>
<p class="MsoNormal"> Transport Layer Security<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Response from server B</b><o:p></o:p></p>
<p class="MsoNormal">Internet Protocol Version 4, Src: 40.68.232.16, Dst: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"> 0100 .... = Version: 4<o:p></o:p></p>
<p class="MsoNormal"> .... 0101 = Header Length: 20 bytes (5)<o:p></o:p></p>
<p class="MsoNormal"> Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)<o:p></o:p></p>
<p class="MsoNormal"> Total Length: 5716<o:p></o:p></p>
<p class="MsoNormal"> Identification: 0x22e1 (8929)<o:p></o:p></p>
<p class="MsoNormal"><b> Flags: 0x4000, Don't fragment<o:p></o:p></b></p>
<p class="MsoNormal"> ...0 0000 0000 0000 = Fragment offset: 0<o:p></o:p></p>
<p class="MsoNormal"> Time to live: 119<o:p></o:p></p>
<p class="MsoNormal"> Protocol: TCP (6)<o:p></o:p></p>
<p class="MsoNormal"> Header checksum: 0xf863 [validation disabled]<o:p></o:p></p>
<p class="MsoNormal"> [Header checksum status: Unverified]<o:p></o:p></p>
<p class="MsoNormal"> Source: 40.68.232.16<o:p></o:p></p>
<p class="MsoNormal"> Destination: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>In response, Ubuntu just fragments the packet anyway and pushes through the WG tunnel<o:p></o:p></b></p>
<p class="MsoNormal">22 0.081525663 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=11, datalen=1408<o:p></o:p></p>
<p class="MsoNormal">23 0.081532164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=12, datalen=1408<o:p></o:p></p>
<p class="MsoNormal">24 0.081535164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=13, datalen=1408<o:p></o:p></p>
<p class="MsoNormal">25 0.081538164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=14, datalen=1408<o:p></o:p></p>
<p class="MsoNormal">26 0.081541064 172.16.21.250 192.168.147.14 WireGuard 410 Transport Data, receiver=0xD8778C02, counter=15, datalen=336<o:p></o:p></p>
<p class="MsoNormal" style="text-align:justify"><span lang="ET" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><span lang="ET" style="color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><b><span lang="EN-GB" style="color:#002060">Sander Saares</span></b><span lang="EN-GB" style="font-size:14.0pt;color:#002060"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="font-size:10.0pt;color:#002060">Advisor
</span><span lang="EN-GB" style="font-size:14.0pt;color:#002060"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="font-size:14.0pt;color:#595959"> </span><span lang="EN-GB" style="font-size:14.0pt;color:#595959"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;color:#7F7F7F">Axinom </span></b><span style="font-size:10.0pt;color:#7F7F7F">| Soola 8 | 51004 Tartu | Estonia</span><span lang="EN-GB" style="font-size:14.0pt;color:#7F7F7F"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#7F7F7F">phone: +49 911 80109-54<o:p></o:p></span></p>
<p class="MsoNormal"><a href="mailto:saares@axinom.com"><span style="font-size:10.0pt;color:#0563C1">saares@axinom.com</span></a><span style="font-size:10.0pt;color:black">
</span><span style="font-size:10.0pt;color:#7F7F7F">| </span><a href="http://www.axinom.com/"><span style="font-size:10.0pt;color:#0563C1">www.axinom.com</span></a><span style="font-size:10.0pt;color:black">
</span><span lang="EN-GB" style="font-size:10.0pt;color:#7F7F7F"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#7F7F7F"> </span><span lang="EN-GB" style="font-size:14.0pt;color:#7F7F7F"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#7F7F7F">Managing Directors: Sergei Gussev, Oleg Knut<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#7F7F7F">Tartu Circuit Court, Reg. 11046287</span><span lang="EN-GB" style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>
[-- Attachment #2: Type: text/plain, Size: 148 bytes --]
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard
^ permalink raw reply [flat|nested] only message in thread
only message in thread, back to index
Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-27 7:55 What would cause Ubuntu 18.04 to perform different fragmentation when forwarding to WireGuard tunnel, depending on packet source? Sander Saares
WireGuard Archive on lore.kernel.org
Archives are clonable:
git clone --mirror https://lore.kernel.org/wireguard/0 wireguard/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 wireguard wireguard/ https://lore.kernel.org/wireguard \
wireguard@lists.zx2c4.com
public-inbox-index wireguard
Example config snippet for mirrors
Newsgroup available over NNTP:
nntp://nntp.lore.kernel.org/com.zx2c4.lists.wireguard
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git