Random packet loss using IPsec with AES128-SHA1

Random packet loss using IPsec with AES128-SHA1

[Gabriele Beltrame]

Hi,

I'm running a few Strongswan 5.3.* CentOS (Kernel 3.16.7, 4.2.6, 4.1.*)
instances on AWS to terminate VPNs between each other and/or to other
devices across the Internet.
While investigating some application issues, I've noticed that on every VPNs
I have random packet losses (from 1% to 4% over 100 to 300 requests sent).
This only happens when the two following conditions are met: (a) AES
encryption used, (b) IP packet size shorter than about (150+8+20)Bytes.

In tcpdump I can actually see all packets (requests and replies) being
"sent" from the router, but on destination server (on the same "LAN") they
are not being received... it's just like if the packet is being lost before
it's being actually serialized onto the network by the XEN NIC driver
Pinging form the vpn router itself always works fine though, never losing a
single packet...

Tested with Kernel 3.16.7, 4.2.6 and a AWS Amazon Linux instance (kernel
4.1.*).
Strongswan and libreswan shows the same issue, so it's not a Strongswan
issue.
Only AES CBS is affected... AES GCM is not affected, furthermore if I use
the Strongswan's kernel-libipsec plugin there is no packet loss.

To recap:
	a. it's not an instance type/size issue (I have the same issue on
everything I've tested with)
	b. it's not a Strongswan issue (I have the same
withlbreswan/Openswan)
	c. it's not a network/related issue (I can actually see on the
router all packets with tcpdump, they are just not received at the
destination (a host on the same network as the "vpn router")
	d. I only see this occurring with AES128 CBS DH2 (and possibly other
key sizes as well)... AES GCM is not affected, as well as 3DES and NULL
encryption
	e. Strongswan developers cannot reproduce the issue on their lab
(Strongswan issue #1220), so possibly there could be something wrong within
the Xen NIC driver
	f. ICMP packets bigger than around 178bytes (150+8+20) seem to not
be affected

Has somebody an idea about where the issue might be and how to
fix/workaround it (I cannot use 3DES and/or AES GCM everywhere and the
Strongswan's kernel-libipsec plugin taxes the CPU a lot more than the kernel
implementation)?

Regards,
Gabriele


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Random packet loss using IPsec with AES128-SHA1

[Steffen Klassert]

On Wed, Dec 16, 2015 at 10:17:54AM +0100, Gabriele Beltrame wrote:
> Hi,
> 
> I'm running a few Strongswan 5.3.* CentOS (Kernel 3.16.7, 4.2.6, 4.1.*)
> instances on AWS to terminate VPNs between each other and/or to other
> devices across the Internet.
> While investigating some application issues, I've noticed that on every VPNs
> I have random packet losses (from 1% to 4% over 100 to 300 requests sent).
> This only happens when the two following conditions are met: (a) AES
> encryption used, (b) IP packet size shorter than about (150+8+20)Bytes.

I've never seen this.

If xfrm statistics are compiled in, a counter is bumped for each packet
dropped by IPsec. You can check these counters in /proc/net/xfrm_stat.

This will tell you at least whether IPsec is the reason for your packet
loss.

RE: Random packet loss using IPsec with AES128-SHA1

[Gabriele Beltrame]

Hi Steffen,

I don't think it's IPsec (I can see the outbound packet in tcpdump), not
alone at least but the XEN /AWS Ethernet driver (or multiple things
concurring to the issue) actually... the odd thing is that it does seem to
affect AES-CBS only (3DES-CBS, AES-GCM are fine)
This is the short discussion on the Strongswan support wiki:
https://wiki.strongswan.org/issues/1220

Thanks,
Gabriele

-----Original Message-----
From: Steffen Klassert [mailto:steffen.klassert@secunet.com] 
Sent: mercoledì 16 dicembre 2015 11:00
To: Gabriele Beltrame <belgab@drvsource.net>
Cc: netdev@vger.kernel.org
Subject: Re: Random packet loss using IPsec with AES128-SHA1

On Wed, Dec 16, 2015 at 10:17:54AM +0100, Gabriele Beltrame wrote:
> Hi,
> 
> I'm running a few Strongswan 5.3.* CentOS (Kernel 3.16.7, 4.2.6, 
> 4.1.*) instances on AWS to terminate VPNs between each other and/or to 
> other devices across the Internet.
> While investigating some application issues, I've noticed that on 
> every VPNs I have random packet losses (from 1% to 4% over 100 to 300
requests sent).
> This only happens when the two following conditions are met: (a) AES 
> encryption used, (b) IP packet size shorter than about (150+8+20)Bytes.

I've never seen this.

If xfrm statistics are compiled in, a counter is bumped for each packet
dropped by IPsec. You can check these counters in /proc/net/xfrm_stat.

This will tell you at least whether IPsec is the reason for your packet
loss.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Random packet loss using IPsec with AES128-SHA1

[Steffen Klassert]

On Thu, Dec 17, 2015 at 04:36:06PM +0100, Gabriele Beltrame wrote:
> Hi Steffen,
> 
> I don't think it's IPsec (I can see the outbound packet in tcpdump), not
> alone at least but the XEN /AWS Ethernet driver (or multiple things
> concurring to the issue) actually.i

Well, in this case you should change the subject to something that
attracts the related driver developers and resend the mail.

Random packet loss using IPsec with AES128-SHA1

[Gabriele Beltrame]

Hi,

I'm running a few Strongswan 5.3.* CentOS (Kernel 3.16.7, 4.2.6, 4.1.*)
instances on AWS to terminate VPNs between each other and/or to other
devices across the Internet.
While investigating some application issues, I've noticed that on every VPNs
I have random packet losses (from 1% to 4% over 100 to 300 requests sent).
This only happens when the two following conditions are met: (a) AES
encryption used, (b) IP packet size shorter than about (150+8+20)Bytes.

In tcpdump I can actually see all packets (requests and replies) being
"sent" from the router, but on destination server (on the same "LAN") they
are not being received... it's just like if the packet is being lost before
it's being actually serialized onto the network by the XEN NIC driver
Pinging form the vpn router itself always works fine though, never losing a
single packet...

Tested with Kernel 3.16.7, 4.2.6 and a AWS Amazon Linux instance (kernel
4.1.*).
Strongswan and libreswan shows the same issue, so it's not a Strongswan
issue.
Only AES CBS is affected... AES GCM is not affected, furthermore if I use
the Strongswan's kernel-libipsec plugin there is no packet loss.

To recap:
	a. it's not an instance type/size issue (I have the same issue on
everything I've tested with)
	b. it's not a Strongswan issue (I have the same
withlbreswan/Openswan)
	c. it's not a network/related issue (I can actually see on the
router all packets with tcpdump, they are just not received at the
destination (a host on the same network as the "vpn router")
	d. I only see this occurring with AES128 CBS DH2 (and possibly other
key sizes as well)... AES GCM is not affected, as well as 3DES and NULL
encryption
	e. Strongswan developers cannot reproduce the issue on their lab
(Strongswan issue #1220), so possibly there could be something wrong within
the Xen NIC driver
	f. ICMP packets bigger than around 178bytes (150+8+20) seem to not
be affected

Has somebody an idea about where the issue might be and how to
fix/workaround it (I cannot use 3DES and/or AES GCM everywhere and the
Strongswan's kernel-libipsec plugin taxes the CPU a lot more than the kernel
implementation)?

Regards,
Gabriele


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus