RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

Vlad Yasevich wrote on 05 February 2007 20:35:

> would you mind terribly, changing the -d "$net" to the
> -i "$net", and run the script with the interface name instead?

I seem to get the same failure when dropping traffic based on interface
as I do when dropping based on address.

> When I block at the ip address, I see the path failover
> in an odd state.  It looks like it happened, but the flow is
> not resumed.  Receive still doesn't get traffic. I think I might

This sounds like it might be the same problem I'm seeing.

My sender is running the 2.6.16.1 kernel with your patch applied, the
receiver is running Fedora Core 6's 2.6.18-1.2798.fc6 kernel.  The
iptables rules are being set on the receiver (so there should be no odd
interactions between the sender's SCTP stack and iptables - as far as
the sender knows the packets have been transmitted and lost in transit).

Thanks.

 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Steve Hill wrote:
> Vlad Yasevich wrote on 05 February 2007 20:35:
> 
>> would you mind terribly, changing the -d "$net" to the
>> -i "$net", and run the script with the interface name instead?
> 
> I seem to get the same failure when dropping traffic based on interface
> as I do when dropping based on address.

Hmm... can you try with a more recent sender please.  Running 2.6.19 or
2.6.20 with my patch, I don't see this problem when a single interface fails.

I see a full path failover withing the 5 second timeout of the table rule.
Once failover happens, the traffic is using the second interface.

I haven't tried forcing the failover back to the first one, but I can
try flip-flopping them and see what happens.

> 
>> When I block at the ip address, I see the path failover
>> in an odd state.  It looks like it happened, but the flow is
>> not resumed.  Receive still doesn't get traffic. I think I might
> 
> This sounds like it might be the same problem I'm seeing.
> 
> My sender is running the 2.6.16.1 kernel with your patch applied, the
> receiver is running Fedora Core 6's 2.6.18-1.2798.fc6 kernel.  The
> iptables rules are being set on the receiver (so there should be no odd
> interactions between the sender's SCTP stack and iptables - as far as
> the sender knows the packets have been transmitted and lost in transit).

Yes, that's what I am doing as well.  I'll see if I can run a more recent
receiver.

Thanks
-vlad

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

Hi Steve

I think I've tracked this down.  Can you apply the attached patch on top
of the one I posted before and re-run your test.

With both patches, I was able flip-flop the downed interface multiple times
and in all cases path failover completed and data flow resumed.

Here is the modified script I was running:

#!/bin/sh

net1="$1"
net2="$2"

flush() {
    iptables -F
    echo "Flush"
    exit
}

trap flush EXIT

while true; do
    # clear table
    iptables -F
    echo "flushed"
    sleep 5

    # block net1
    iptables -A INPUT -i "$net1" -p sctp -j DROP
    echo "set net1"
    sleep 5

    # clear table
    iptables -F
    echo "flushed"
    sleep 5

    # block net2
    iptables -A INPUT -i "$net2" -p sctp -j DROP
    echo "set net2"
    sleep 5
done

I was able to run this script for 10 minutes sustaining the message flow.

-vlad

[-- Attachment #2: 0001-SCTP-Strike-the-transport-before-updating-rto.patch --]
[-- Type: text/x-patch, Size: 1322 bytes --]

>From 72d6856f7e45a17e0910e0eacd1a01d44fafd1c0 Mon Sep 17 00:00:00 2001
From: Vlad Yasevich <vladislav.yasevich@hp.com>
Date: Wed, 7 Feb 2007 14:58:25 -0500
Subject: [PATCH] [SCTP] Strike the transport before updating rto

Once we reach a poing where we exceed the max.path.retrans
strike the trasport before updating the rto.  This will force
transport switch at the right time, instead of 1 retransmit too
late.

Signed-off-by: Vlad Yasevich <vladislav.yasevich@hp.com>
---
 net/sctp/sm_statefuns.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c
index fbbc9e6..801f9d6 100644
--- a/net/sctp/sm_statefuns.c
+++ b/net/sctp/sm_statefuns.c
@@ -4605,12 +4605,12 @@ sctp_disposition_t sctp_sf_do_6_3_3_rtx(const struct sctp_endpoint *ep,
 	 * sent as soon as cwnd allows (normally when a SACK arrives).
 	 */
 
-	/* NB: Rules E4 and F1 are implicit in R1.  */
-	sctp_add_cmd_sf(commands, SCTP_CMD_RETRAN, SCTP_TRANSPORT(transport));
-
 	/* Do some failure management (Section 8.2). */
 	sctp_add_cmd_sf(commands, SCTP_CMD_STRIKE, SCTP_TRANSPORT(transport));
 
+	/* NB: Rules E4 and F1 are implicit in R1.  */
+	sctp_add_cmd_sf(commands, SCTP_CMD_RETRAN, SCTP_TRANSPORT(transport));
+
 	return SCTP_DISPOSITION_CONSUME;
 }
 
-- 
1.5.0.rc3.g6506

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

On Wed, 7 Feb 2007, Vlad Yasevich wrote:

> I think I've tracked this down.  Can you apply the attached patch on top
> of the one I posted before and re-run your test.

Using the 2.6.20 kernel on the sending side with both patches applied, the
problem seems to be fixed.

Thanks.

-- 
 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Steve Hill wrote:
> On Wed, 7 Feb 2007, Vlad Yasevich wrote:
> 
>> I think I've tracked this down.  Can you apply the attached patch on top
>> of the one I posted before and re-run your test.
> 
> Using the 2.6.20 kernel on the sending side with both patches applied, the
> problem seems to be fixed.
> 
> Thanks.
> 

Excellent.  I'll push out the new ones to Sridhar today.

-vlad

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Hi Steve

would you mind terribly, changing the -d "$net" to the 
-i "$net", and run the script with the interface name instead?

The reason is, that I see 2 different behaviors between blocking
by interface and blocking by IP and would like to find out if
you see it too.

When I block at the interface ( -i eth1 in my case), I see
the path failover happen and flow if resumed.

When I block at the ip address, I see the path failover
in an odd state.  It looks like it happened, but the flow is
not resumed.  Receive still doesn't get traffic. I think I might
be running a buggy receiver, but I am not 100% sure.  In my
case, the sender if running 2.6.10-rc7 and receiver is running
Ubuntu 2.6.17-10.

I'll try running against a different receiver as well.

-vlad

Steve Hill wrote:
> Vlad Yasevich wrote on 05 February 2007 17:08:
> 
>>   1. What did you set the sinfo_timetolive to?
> 
> I presume you mean the timetolive parameter of sctp_sendmsg()? - this
> was set to 1400ms (as previously mentioned, this was in error but it
> does appear to have highlighted a problem with the stack itself).
> 
>>   2. What specific netfilter rule to do use to simulate
>> network outage?
>>      I was using '-t filter -A INPUT -i eth0 -p sctp -j DROP'
> 
> iptables -A INPUT -d 192.168.2.0/24 -p sctp -j DROP
> 
>> Just trying to get more info to simulate this.  My prior attempts
>> recovered quickly with my patch.
> 
> I usually (but not always - sometimes it happens on the first attempt)
> have to add and remove the iptables rule a few times while running
> traffic over the association in order to reproduce the problem.  I'm
> running traffic at a rate of around 500 data chunks per second.  Each
> data chunk has a 44 octet payload.
> 
> The script I'm using to toggle the iptables rule is below:
> 
> ----------
> #!/bin/sh
> 
> net="$1"
> 
> flush() {
> 	iptables -F
> 	echo "Flush"
> 	exit
> }
> 
> trap flush EXIT
> 
> while true; do
> 	iptables -A INPUT -d "$net" -p sctp -j DROP
> 	echo "set"
> 	sleep 5
> 	iptables -F
> 	echo "flushed"
> 	sleep 5
> done 
> ----------
> 
>  - Steve Hill
>    Software Engineer
>    Dialogic
>    Fordingbridge, Hampshire, UK
>    +44-1425-651392
>    steve.hill@dialogic.com
> 

RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

Vlad Yasevich wrote on 05 February 2007 17:08:

>   1. What did you set the sinfo_timetolive to?

I presume you mean the timetolive parameter of sctp_sendmsg()? - this
was set to 1400ms (as previously mentioned, this was in error but it
does appear to have highlighted a problem with the stack itself).

>   2. What specific netfilter rule to do use to simulate
> network outage?
>      I was using '-t filter -A INPUT -i eth0 -p sctp -j DROP'

iptables -A INPUT -d 192.168.2.0/24 -p sctp -j DROP

> Just trying to get more info to simulate this.  My prior attempts
> recovered quickly with my patch.

I usually (but not always - sometimes it happens on the first attempt)
have to add and remove the iptables rule a few times while running
traffic over the association in order to reproduce the problem.  I'm
running traffic at a rate of around 500 data chunks per second.  Each
data chunk has a 44 octet payload.

The script I'm using to toggle the iptables rule is below:

----------
#!/bin/sh

net="$1"

flush() {
	iptables -F
	echo "Flush"
	exit
}

trap flush EXIT

while true; do
	iptables -A INPUT -d "$net" -p sctp -j DROP
	echo "set"
	sleep 5
	iptables -F
	echo "flushed"
	sleep 5
done 
----------

 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Hi Steve

Steve Hill wrote:
> Vlad Yasevich wrote on 05 February 2007 16:39:
> 
>> Once you start simulating the network failure, how long do you wait?
>>
>> If you have not changed rto_max and path_max_retrans, you can end up
>> waiting quite a while for the full path switchover.  This will also
>> severely slow down your retransmissions...
> 
> I'm waiting several minutes, and I'm using setsockopt() to set the
> timeouts fairly low:
> 	srto_initial = 1000ms
> 	srto_min = 200ms
> 	srto_max = 1400ms
> 	spp_pathmaxrxt = 2
> 	spp_hbinterval = 1000ms

Ok, thanks for the info.  Two more questions:

  1. What did you set the sinfo_timetolive to?
  2. What specific netfilter rule to do use to simulate network outage?
     I was using '-t filter -A INPUT -i eth0 -p sctp -j DROP'

Just trying to get more info to simulate this.  My prior attempts 
recovered quickly with my patch.

Thanks
-vlad

> 
> The network's round trip time (indicated by ping) is on the order of
> <0.25ms.
> 
>  - Steve Hill
>    Software Engineer
>    Dialogic
>    Fordingbridge, Hampshire, UK
>    +44-1425-651392
>    steve.hill@dialogic.com
> 

RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

Vlad Yasevich wrote on 05 February 2007 16:39:

> Once you start simulating the network failure, how long do you wait?
> 
> If you have not changed rto_max and path_max_retrans, you can end up
> waiting quite a while for the full path switchover.  This will also
> severely slow down your retransmissions...

I'm waiting several minutes, and I'm using setsockopt() to set the
timeouts fairly low:
	srto_initial = 1000ms
	srto_min = 200ms
	srto_max = 1400ms
	spp_pathmaxrxt = 2
	spp_hbinterval = 1000ms

The network's round trip time (indicated by ping) is on the order of
<0.25ms.

 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

Steve Hill wrote:
> Vlad Yasevich wrote on 25 January 2007 16:33:
> 
>> Can you try the attached patch and let me know if the problem is
>> fixed.  You can try reducing rto_max or path_max_retrans to get the
>> failover to happen a little faster. 
> 
> Sorry for the delay - I've been on vacation for the past week.
> 
> I've tried applying the patch.  However, the failure still seems to
> happen in the original test system with a patched kernel.  A look at the
> network traffic shows that the receiving side is still returning a
> gap-ack, the chunks in the gap are never resent and I don't see a
> FORWARD TSN for them.

Once you start simulating the network failure, how long do you wait?

If you have not changed rto_max and path_max_retrans, you can end up
waiting quite a while for the full path switchover.  This will also
severely slow down your retransmissions...

Try reducing /proc/sys/net/sctp/rto_max to say 6000ms (6 seconds).  It
defaults to (60000 ms or 1 minute).  That reduction means that the
retransmit timeout will max out at 6 seconds.

The problem is that you will not see the FORWARD TSN on the alternate
path until the original path is marked down.  That requires us to
exceed the path_max_retrans counter
(/proc/sys/net/sctp/path_max_retrans).  This means that you will see
the gap grow until a time when the rto exceeds the data expiry time.
At this pint, you will not see any new data sent, since it will expire
before being sent.  So, it will look to you like nothing is happening.

The gap will be closed once the FWD-TSN is transmitted over the
alternate transport.

-vlad

> 
>  - Steve Hill
>    Software Engineer
>    Dialogic
>    Fordingbridge, Hampshire, UK
>    +44-1425-651392
>    steve.hill@dialogic.com
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Lksctp-developers mailing list
> Lksctp-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lksctp-developers
> 

RE: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Steve Hill]

Vlad Yasevich wrote on 25 January 2007 16:33:

> Can you try the attached patch and let me know if the problem is
> fixed.  You can try reducing rto_max or path_max_retrans to get the
> failover to happen a little faster. 

Sorry for the delay - I've been on vacation for the past week.

I've tried applying the patch.  However, the failure still seems to
happen in the original test system with a patched kernel.  A look at the
network traffic shows that the receiving side is still returning a
gap-ack, the chunks in the gap are never resent and I don't see a
FORWARD TSN for them.

 - Steve Hill
   Software Engineer
   Dialogic
   Fordingbridge, Hampshire, UK
   +44-1425-651392
   steve.hill@dialogic.com

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

BTW, if anyone needs a reproducer, I can provide one.

-vlad

Re: [Lksctp-developers] Fw: Intermittent SCTP multihoming breakage

[Vlad Yasevich]

[-- Attachment #1: Type: text/plain, Size: 1190 bytes --]

Hi Steve

Steve Hill wrote:
> On Wed, 10 Jan 2007, Sridhar Samudrala wrote:
> 
>> So looks like there may be an issue with PR-SCTP(partial reliability)
>> support and packet loss. I will take a look into this.
>>
>> Do you still see this problem even if you don't set timetolive?
> 
> No, the problem seems to go away if the timetolive is set to 0, so this is
> what I have now done since I had not intended to set the timetolive in the
> first place (but I thought it was still worth posting details of the
> problem since it does appear to be a bug).
> 

I think I found this bug.  It was rather interesting to figure out.  The problem
appears to be that data messages time-out within the rto.  As a result, they
move the abandoned list and are never retransmitted.  This clears the retransmit
list and the retransmit timer, however the data is still charged as in-flight against
the association.  This in turn causes new data not to be send, since we are 'supposedly'
utilizing our congestion window.

Can you try the attached patch and let me know if the problem is fixed.  You can 
try reducing rto_max or path_max_retrans to get the failover to happen a little faster.

Regards
-vlad

[-- Attachment #2: 0001-SCTP-Fix-connection-hang-slowdown-with-PR-SCTP.txt --]
[-- Type: text/plain, Size: 2804 bytes --]

[SCTP]: Fix connection hang with PR-SCTP

The problem that this patch corrects happens when all
of the following conditions are satisfisfied:
    1.  PR-SCTP is used and the timeout on the chunks is
        set below RTO.Max.
    2.  One of the paths on a multihomed associations
        is brought down.

In this scenario, data will expire within the rto of the
initial transmission and will never be retransmitted.  However
this data still fills the send buffer and is counted against
the association as outstanding data.  This causes any new
data to not be sent and retransmission to not happen.

The fix is to discount the abandoned data from the outstanding
count and peers rwnd estimation.  This allows new data to
be sent and a retransmission timer restarted.  Even though
this new data will most like expire withing the rto, the
timer still counts as a strike agains the transport and forces
the FORWARD-TSN chunk to be retransmitted as well.

Signed-off-by: Vlad Yasevich <vladislav.yasevich@hp.com>
---
 net/sctp/outqueue.c |   27 ++++++++++++++++++++++-----
 1 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/net/sctp/outqueue.c b/net/sctp/outqueue.c
index fba567a..54d1b7f 100644
--- a/net/sctp/outqueue.c
+++ b/net/sctp/outqueue.c
@@ -396,6 +396,19 @@ void sctp_retransmit_mark(struct sctp_outq *q,
 		if (sctp_chunk_abandoned(chunk)) {
 			list_del_init(lchunk);
 			sctp_insert_list(&q->abandoned, lchunk);
+
+			/* If this chunk has not been previousely acked,
+			 * stop considering it 'outstanding'.  Our peer
+			 * will most likely never see it since it will
+			 * not be retransmitted
+			 */
+			if (!chunk->tsn_gap_acked) {
+				chunk->transport->flight_size -=
+						sctp_data_size(chunk);
+				q->outstanding_bytes -= sctp_data_size(chunk);
+				q->asoc->peer.rwnd += (sctp_data_size(chunk) +
+							sizeof(struct sk_buff));
+			}
 			continue;
 		}
 
@@ -1244,6 +1257,15 @@ static void sctp_check_transmitted(struct sctp_outq *q,
 		if (sctp_chunk_abandoned(tchunk)) {
 			/* Move the chunk to abandoned list. */
 			sctp_insert_list(&q->abandoned, lchunk);
+
+			/* If this chunk has not been acked, stop
+			 * considering it as 'outstanding'.
+			 */
+			if (!tchunk->tsn_gap_acked) {
+				tchunk->transport->flight_size -=
+						sctp_data_size(tchunk);
+				q->outstanding_bytes -= sctp_data_size(tchunk);
+			}
 			continue;
 		}
 
@@ -1695,11 +1717,6 @@ static void sctp_generate_fwdtsn(struct sctp_outq *q, __u32 ctsn)
 		 */ 
 		if (TSN_lte(tsn, ctsn)) {
 			list_del_init(lchunk);
-			if (!chunk->tsn_gap_acked) {
-				chunk->transport->flight_size -=
-					sctp_data_size(chunk);
-				q->outstanding_bytes -= sctp_data_size(chunk);
-			}
 			sctp_chunk_free(chunk);
 		} else {
 			if (TSN_lte(tsn, asoc->adv_peer_ack_point+1)) {
-- 
1.4.4.2.g8336