[PATCH V2 net-next 0/3] net: implement SMC-R solution

[PATCH V2 net-next 0/3] net: implement SMC-R solution

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

Eric,

this is V2 of my SMC-R patches, containing especially a new version of the
required tcp changes. As you suggested, SMC-specific hooks in the TCP-code
are built only for CONFIG_AFSMC. And I come up with helpers in include files
to avoid spreading net #ifdef in C files.

V2 changes:
1. activate tcp changes for CONFIG_AFSMC only (as suggested by Eric Dumazet)
2. add additional hook in net/core/sock.c
3. fix bitfield endianness problem

Thanks,
        Ursula

In 2013, IBM introduced an optimized communications solution for the
IBM zEnterprise EC12 and BC12 (s390 in Linux terminology) that is
comprised of the IBM 10GbE RoCE Express feature with Shared Memory
Communications-RDMA (SMC-R) protocol [1].
SMC-R is designed for the enterprise data center environment and is an open
protocol as specified in the informational RFC [2]. The final draft
submitted by IBM has been approved for publication and is in the final
editorial stage. Another implementation of this protocol is available since
2013 with IBM z/OS Version 2 Release 1. 

SMC-R provides a “sockets over RDMA” solution that leverages industry
standard RDMA over Converged Ethernet (RoCE) technology.

IBM has developed a Linux implementation of the SMC-R standard. A new
socket protocol family AF_SMC is introduced. A preload library can be used
to enable TCP-based applications to use SMC-R without changes. 

Key aspects of SMC-R are: 
1. Provides optimized performance compared to standard TCP/IP over Ethernet
   within the data center for both request/response (latency) and streaming
   workloads (CPU savings) [3]. 
   Initial benchmarks on Linux on x86 processors have shown latency
   reduction of up to 52% with a throughput gain of 111% using SMC-R vs TCP
   for request/response message patterns (10 concurrent TCP connections
   with 16KB    messages) and CPU savings of up to 69% for streaming data
   patterns (single TCP connection with 20MB of data in one direction).
   [1] is currently updated to contain more detailed information on Linux
   and performance.
2. In order to preserve the traditional network administrative model the
   SMC-R protocol ties into the existing IP addresses and uses TCP's
   handshake to establish connections. This allows existing management
   tools and security infrastructure to control the creation of SMC
   connections.
3. The SMC-R protocol logically bonds multiple RoCE adapters together
   providing    redundancy with transparent fail-over for improved high
   availability, increased bandwidth and load balancing across multiple
   RDMA-capable devices.
4. Due to its handshake protocol, SMC-R is compatible with (transparent to)
   existing TCP connection load balancers that are commonly used in the
   enterprise data center environment for multi-tier application workloads.
5. SMC-R's handshake protocol allows for transparent fallback to TCP/IP,
   should one of the peers not be capable of the protocol.

Additional SMC-R overview and reference materials are available [1].  

The SMC-R “rendezvous" protocol eliminates the need for RDMA-CM and the
exchange occurs through an initial TCP connection. Building on a TCP
connection to establish an SMC-R connection solves many key requirements,
including #4 and #5 above.
The rendezvous process occurs in 2 phases: 
1. TCP/IP 3-way exchange:
   Initiated when both client and server indicate SMC-R capability by
   including TCP experimental options on the TCP/IP 3-way handshake (syn
   flows) as described in RFC6994 [4]. The ExID assigned by IANA is
   0xE2D4C3D9 [5]. 
2. SMC-R 3-way exchange:
   When both partners indicate SMC-R capability then at the completion of
   the 3-way TCP handshake the SMC-R layers in each peer take control of
   the TCP connection and exchange their RDMA credentials. If this 3-way
   exchange    completes successfully the connection continues using SMC-R.
   If the exchange is not successful the connections falls back to standard
   TCP/IP. 

References:
[1] SMC-R Overview and Reference Materials:
    http://www-01.ibm.com/software/network/commserver/SMCR/ 
[2] SMC-R Informational RFC:
    http://tools.ietf.org/html/draft-fox-tcpm-shared-memory-rdma-07
[3] Linux SMC-R Overview and Performance Summary
    (archs x86 and s390):
    http://www-01.ibm.com/software/network/commserver/SMCR/ 
[4] Shared Use of TCP Experimental Options RFC 6994:
    https://tools.ietf.org/rfc/rfc6994.txt    
[5] IANA ExID SMCR: 
    http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xhtml#tcp-exids

The patch series is prepared to apply to net-next and consists of these
parts:
1. net/ipv4/tcp: TCP experimental option
2. net: definitions to establish new socket family
3. net/smc: new socket family

In the future, SMC-R will be enhanced to cover:
- IPv6 support
- Tracing
- Statistics support

Ursula Braun (3):
  tcp: introduce TCP experimental option for SMC
  net: introduce socket family constants
  smc: introduce socket family AF_SMC

 include/linux/socket.h     |    4 +-
 include/linux/tcp.h        |   16 +-
 include/net/request_sock.h |    3 +-
 include/net/smc.h          |   13 +
 include/net/tcp.h          |  145 ++
 net/Kconfig                |    1 +
 net/Makefile               |    1 +
 net/core/sock.c            |   15 +-
 net/ipv4/tcp_input.c       |    8 +
 net/ipv4/tcp_minisocks.c   |    3 +
 net/ipv4/tcp_output.c      |   23 +-
 net/smc/Kconfig            |    9 +
 net/smc/Makefile           |    3 +
 net/smc/af_smc.c           | 3142 ++++++++++++++++++++++++++++++++++++++++++
 net/smc/af_smc.h           |  706 ++++++++++
 net/smc/smc_core.c         | 3291 ++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_llc.c          | 1597 +++++++++++++++++++++
 net/smc/smc_llc.h          |  192 +++
 net/smc/smc_proc.c         |  884 ++++++++++++
 19 files changed, 10034 insertions(+), 22 deletions(-)
 create mode 100644 include/net/smc.h
 create mode 100644 net/smc/Kconfig
 create mode 100644 net/smc/Makefile
 create mode 100644 net/smc/af_smc.c
 create mode 100644 net/smc/af_smc.h
 create mode 100644 net/smc/smc_core.c
 create mode 100644 net/smc/smc_llc.c
 create mode 100644 net/smc/smc_llc.h
 create mode 100644 net/smc/smc_proc.c

-- 
2.3.8

[PATCH V2 net-next 1/3] tcp: introduce TCP experimental option for SMC

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

The SMC-R protocol defines dynamic discovery of peers. This is done by
implementing experimental TCP options as defined in RFC6994. The TCP code
needs to be extended to support RFC6994.

Setting the TCP experimental option for SMC-R [2] will be triggered from
kernel exploiters like the new SMC-R socket family by setting a new
flag "syn_smc" on struct tcp_sock of the connecting and the listening
socket. If the client peer is SMC-R capable, flag syn_smc is kept on the
connecting socket after the 3-way TCP handshake, otherwise it is reset.
If the server peer is SMC-R capable, the new connected TCP socket has
the new flag set, otherwise not.

Code snippet client:
  tcp_sk(sock->sk)->syn_smc = 1;
  rc = kernel_connect(sock, addr, alen, flags);
  if (tcp_sk(sock->sk)->syn_smc) {
          /* switch to smc for this connection */

Code snippet server:
  tcp_sk(sock->sk)->syn_smc = 1;
  rc = kernel_listen(sock, backlog);
  rc = kernel_accept(sock, &newsock, 0);
  if (tcp_sk(newsock->sk)->syn_smc) {
          /* switch to smc for this connection */

References:
[1] Shared Use of TCP Experimental Options RFC 6994:
    https://tools.ietf.org/rfc/rfc6994.txt    
[2] IANA ExID SMCR: 
    http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xhtml#tcp-exids

This patch has already been posted in June 2013, but Dave Miller has
postponed applying till the user of the new flags, ie. the entire SMC-R
protocol stack is implemented.

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 include/linux/tcp.h        |  16 ++++-
 include/net/request_sock.h |   3 +-
 include/net/tcp.h          | 145 +++++++++++++++++++++++++++++++++++++++++++++
 net/ipv4/tcp_input.c       |   8 +++
 net/ipv4/tcp_minisocks.c   |   3 +
 net/ipv4/tcp_output.c      |  23 ++-----
 6 files changed, 179 insertions(+), 19 deletions(-)

diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 48c3696..1b9a698 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -90,15 +90,28 @@ struct tcp_options_received {
 		sack_ok : 4,	/* SACK seen on SYN packet		*/
 		snd_wscale : 4,	/* Window scaling received from sender	*/
 		rcv_wscale : 4;	/* Window scaling to send to receiver	*/
+	u8	smc_capability:1; /* SMC capability			*/
 	u8	num_sacks;	/* Number of SACK blocks		*/
 	u16	user_mss;	/* mss requested by user in ioctl	*/
 	u16	mss_clamp;	/* Maximal mss, negotiated at connection setup */
 };
 
+#if IS_ENABLED(CONFIG_AFSMC)
+static inline void smc_clear_rx_opt(struct tcp_options_received *rx_opt)
+{
+	rx_opt->smc_capability = 0;
+}
+#else
+static inline void smc_clear_rx_opt(struct tcp_options_received *rx_opt)
+{
+}
+#endif
+
 static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 {
 	rx_opt->tstamp_ok = rx_opt->sack_ok = 0;
 	rx_opt->wscale_ok = rx_opt->snd_wscale = 0;
+	smc_clear_rx_opt(rx_opt);
 }
 
 /* This is the max number of SACKS that we'll generate and process. It's safe
@@ -207,7 +220,8 @@ struct tcp_sock {
 		syn_fastopen_exp:1,/* SYN includes Fast Open exp. option */
 		syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
 		save_syn:1,	/* Save headers of SYN packet */
-		is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
+		is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */
+		syn_smc:1;	/* SYN includes SMC			*/
 	u32	tlp_high_seq;	/* snd_nxt at the time of TLP retransmit. */
 
 /* RTT measurement */
diff --git a/include/net/request_sock.h b/include/net/request_sock.h
index 87935ca..dee47d2 100644
--- a/include/net/request_sock.h
+++ b/include/net/request_sock.h
@@ -55,7 +55,8 @@ struct request_sock {
 	struct sock			*rsk_listener;
 	u16				mss;
 	u8				num_retrans; /* number of retransmits */
-	u8				cookie_ts:1; /* syncookie: encode tcpopts in timestamp */
+	u8				cookie_ts:1, /* syncookie: encode tcpopts in timestamp */
+					smc_capability:1;
 	u8				num_timeout:7; /* number of timeouts */
 	/* The following two fields can be easily recomputed I think -AK */
 	u32				window_clamp; /* window clamp at creation time */
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 364426a..b51a6c1 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -46,6 +46,7 @@
 
 #include <linux/seq_file.h>
 #include <linux/memcontrol.h>
+#include <asm/unaligned.h>
 
 extern struct inet_hashinfo tcp_hashinfo;
 
@@ -185,6 +186,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
  * experimental options. See draft-ietf-tcpm-experimental-options-00.txt
  */
 #define TCPOPT_FASTOPEN_MAGIC	0xF989
+#define TCPOPT_SMC_MAGIC	0xE2D4C3D9
 
 /*
  *     TCP option lengths
@@ -197,6 +199,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_MD5SIG         18
 #define TCPOLEN_FASTOPEN_BASE  2
 #define TCPOLEN_EXP_FASTOPEN_BASE  4
+#define TCPOLEN_EXP_SMC_BASE   6
 
 /* But this is what stacks really send out. */
 #define TCPOLEN_TSTAMP_ALIGNED		12
@@ -207,6 +210,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_SACK_PERBLOCK		8
 #define TCPOLEN_MD5SIG_ALIGNED		20
 #define TCPOLEN_MSS_ALIGNED		4
+#define TCPOLEN_EXP_SMC_BASE_ALIGNED    8
 
 /* Flags in tp->nonagle */
 #define TCP_NAGLE_OFF		1	/* Nagle's algo is disabled */
@@ -1762,4 +1766,145 @@ static inline void skb_set_tcp_pure_ack(struct sk_buff *skb)
 	skb->truesize = 2;
 }
 
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
+#if IS_ENABLED(CONFIG_AFSMC)
+static inline void smc_parse_options(const struct tcphdr *th,
+				     struct tcp_options_received *opt_rx,
+				     const unsigned char *ptr,
+				     int opsize)
+{
+	if (th->syn && !(opsize & 1) &&
+	    opsize >= TCPOLEN_EXP_SMC_BASE &&
+	    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC)
+		opt_rx->smc_capability = 1;
+}
+
+static inline void smc_options_write(__be32 *ptr, u16 *options)
+{
+	if (unlikely(OPTION_SMC & *options)) {
+		*ptr++ = htonl((TCPOPT_NOP  << 24) |
+			       (TCPOPT_NOP  << 16) |
+			       (TCPOPT_EXP <<  8) |
+			       (TCPOLEN_EXP_SMC_BASE));
+		*ptr++ = htonl(TCPOPT_SMC_MAGIC);
+	}
+}
+
+static inline void smc_set_option(struct tcp_sock *tp,
+				  struct tcp_out_options *opts,
+				  unsigned int *remaining)
+{
+	if (tp->syn_smc) {
+		u32 need = TCPOLEN_EXP_SMC_BASE_ALIGNED;
+
+		if (*remaining >= need) {
+			opts->options |= OPTION_SMC;
+			*remaining -= need;
+		}
+	}
+}
+static inline void smc_set_option_cond(struct tcp_sock *tp,
+				       struct request_sock *req,
+				       struct tcp_out_options *opts,
+				       unsigned int *remaining)
+{
+	if (tp->syn_smc && req->smc_capability) {
+		u32 need = TCPOLEN_EXP_SMC_BASE_ALIGNED;
+
+		if (*remaining >= need) {
+			opts->options |= OPTION_SMC;
+			*remaining -= need;
+		}
+	}
+}
+
+static inline void smc_set_capability(struct request_sock *req,
+				      struct tcp_options_received *rx_opt)
+{
+	if (rx_opt->smc_capability)
+		req->smc_capability = 1;
+}
+
+static inline void smc_reset_capability(struct request_sock *req)
+{
+	req->smc_capability = 0;
+}
+
+static inline void smc_check_reset_syn(struct tcp_sock *tp)
+{
+	if (tp->syn_smc && !tp->rx_opt.smc_capability)
+		tp->syn_smc = 0;
+}
+
+static inline void smc_check_reset_syn_req(struct tcp_sock *oldtp,
+					   struct request_sock *req,
+					   struct tcp_sock *newtp)
+{
+	if (oldtp->syn_smc && !req->smc_capability)
+		newtp->syn_smc = 0;
+}
+
+#else
+static inline void smc_parse_options(const struct tcphdr *th,
+				     struct tcp_options_received *opt_rx,
+				     const unsigned char *ptr,
+				     int opsize)
+{
+}
+
+static inline void smc_options_write(__be32 *ptr, u16 *options)
+{
+}
+
+static inline void smc_set_option(struct tcp_sock *tp,
+				  struct tcp_out_options *opts,
+				  unsigned int *remaining)
+{
+}
+
+static inline void smc_set_option_cond(struct tcp_sock *tp,
+				       struct request_sock *req,
+				       struct tcp_out_options *opts,
+				       unsigned int *remaining)
+{
+}
+
+static inline void smc_set_capability(struct request_sock *req,
+				      struct tcp_options_received *rx_opt)
+{
+}
+
+static inline void smc_reset_capability(struct request_sock *req)
+{
+}
+
+static inline void smc_check_reset_syn(struct tcp_sock *tp)
+{
+}
+
+static inline void smc_check_reset_syn_req(struct tcp_sock *oldtp,
+					   struct request_sock *req,
+					   struct tcp_sock *tp)
+{
+}
+
+#endif /* IS_ENABLED(CONFIG_AFSMC) */
+
 #endif	/* _TCP_H */
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 1578fc2..9b49240 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -3748,6 +3748,9 @@ void tcp_parse_options(const struct sk_buff *skb,
 					tcp_parse_fastopen_option(opsize -
 						TCPOLEN_EXP_FASTOPEN_BASE,
 						ptr + 2, th->syn, foc, true);
+				else
+					smc_parse_options(th, opt_rx, ptr,
+							  opsize);
 				break;
 
 			}
@@ -5556,6 +5559,8 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		 * is initialized. */
 		tp->copied_seq = tp->rcv_nxt;
 
+		smc_check_reset_syn(tp);
+
 		smp_mb();
 
 		tcp_finish_connect(sk, skb);
@@ -6002,6 +6007,7 @@ static void tcp_openreq_init(struct request_sock *req,
 
 	req->rcv_wnd = 0;		/* So that tcp_send_synack() knows! */
 	req->cookie_ts = 0;
+	smc_reset_capability(req);
 	tcp_rsk(req)->rcv_isn = TCP_SKB_CB(skb)->seq;
 	tcp_rsk(req)->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
 	tcp_rsk(req)->snt_synack = tcp_time_stamp;
@@ -6142,6 +6148,8 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 	/* Note: tcp_v6_init_req() might override ir_iif for link locals */
 	inet_rsk(req)->ir_iif = sk->sk_bound_dev_if;
 
+	smc_set_capability(req, &tmp_opt);
+
 	af_ops->init_req(req, sk, skb);
 
 	if (security_inet_conn_request(sk, skb, req))
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 6d8795b..62e6c2c 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -443,6 +443,9 @@ struct sock *tcp_create_openreq_child(struct sock *sk, struct request_sock *req,
 		struct tcp_request_sock *treq = tcp_rsk(req);
 		struct inet_connection_sock *newicsk = inet_csk(newsk);
 		struct tcp_sock *newtp = tcp_sk(newsk);
+		struct tcp_sock *oldtp = tcp_sk(sk);
+
+		smc_check_reset_syn_req(oldtp, req, newtp);
 
 		/* Now setup tcp_sock */
 		newtp->pred_flags = 0;
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 7105784..17ddabd 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -419,23 +419,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
  *
  * Beware: Something in the Internet is very sensitive to the ordering of
@@ -542,6 +525,8 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 		}
 		ptr += (len + 3) >> 2;
 	}
+
+	smc_options_write(ptr, &options);
 }
 
 /* Compute TCP options for SYN packets. This is not the final
@@ -609,6 +594,8 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	smc_set_option(tp, opts, &remaining);
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -670,6 +657,8 @@ static unsigned int tcp_synack_options(struct sock *sk,
 		}
 	}
 
+	smc_set_option_cond(tcp_sk(sk), req, opts, &remaining);
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
-- 
2.3.8

Re: [PATCH V2 net-next 1/3] tcp: introduce TCP experimental option for SMC

[David Miller]

From: Ursula Braun <ubraun@linux.vnet.ibm.com>
Date: Tue, 14 Jul 2015 14:42:33 +0200

> diff --git a/include/linux/tcp.h b/include/linux/tcp.h
> index 48c3696..1b9a698 100644
> --- a/include/linux/tcp.h
> +++ b/include/linux/tcp.h
> @@ -90,15 +90,28 @@ struct tcp_options_received {
>  		sack_ok : 4,	/* SACK seen on SYN packet		*/
>  		snd_wscale : 4,	/* Window scaling received from sender	*/
>  		rcv_wscale : 4;	/* Window scaling to send to receiver	*/
> +	u8	smc_capability:1; /* SMC capability			*/
>  	u8	num_sacks;	/* Number of SACK blocks		*/
>  	u16	user_mss;	/* mss requested by user in ioctl	*/
>  	u16	mss_clamp;	/* Maximal mss, negotiated at connection setup */
>  };

This adds new space to this structure, which can be avoided.

sack_ok only actually needs 3 bits, not 4, then you can use the extra
bit for smc_capability.

> diff --git a/include/net/request_sock.h b/include/net/request_sock.h
> index 87935ca..dee47d2 100644
> --- a/include/net/request_sock.h
> +++ b/include/net/request_sock.h
> @@ -55,7 +55,8 @@ struct request_sock {
>  	struct sock			*rsk_listener;
>  	u16				mss;
>  	u8				num_retrans; /* number of retransmits */
> -	u8				cookie_ts:1; /* syncookie: encode tcpopts in timestamp */
> +	u8				cookie_ts:1, /* syncookie: encode tcpopts in timestamp */
> +					smc_capability:1;
>  	u8				num_timeout:7; /* number of timeouts */
>  	/* The following two fields can be easily recomputed I think -AK */
>  	u32				window_clamp; /* window clamp at creation time */

Again, similar situation here.

Please find a way to add your new boolean value without expanding the size of
this structure.  Here is it clear that a single u8 is fully consumed by the
cookie_ts and num_timeout bit fields.

> +#if IS_ENABLED(CONFIG_AFSMC)

I think this ifdef is completely pointless.

What do you think every Linux distribution is going to do for their kernels?
They are going to turn everything on.

So you need to find a way for your new feature to be nearly zero cost,
especially in the fast paths, assuming the code is enabled.

[PATCH V3 net-next 0/5] net: implement SMC-R solution

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

Dave,

this is V3 of my SMC-R patches, containing mainly a new version of the
required tcp changes. Dave, you requested the new feature to be nearly
zero cost. Thus my approach is a technique using Static Keys. Do you
basicly agree with such an approach? 
For now I kept the CONFIG_AFSMC option in addition. Do you prefer to
get rid of it in the tcp-code?

V3 changes:
1. Avoid adding of new space for smc-related bits in the tcp structures.
2. Make the smc feature to be nearly zero cost using Static Keys / jump
   labels
3. Increase / decrease smc static key in the smc-code
4. Make sure the next-to-last patch does not break the build
5. Additional pnet table checking

V2 changes:
1. activate tcp changes for CONFIG_AFSMC only (as suggested by Eric Dumazet)
2. add additional hook in net/core/sock.c
3. fix bitfield endianness problem

Thanks,
        Ursula

In 2013, IBM introduced an optimized communications solution for the
IBM zEnterprise EC12 and BC12 (s390 in Linux terminology) that is
comprised of the IBM 10GbE RoCE Express feature with Shared Memory
Communications-RDMA (SMC-R) protocol [1].
SMC-R is designed for the enterprise data center environment and is an open
protocol as specified in the informational RFC [2]. The final draft
submitted by IBM has been approved for publication and is in the final
editorial stage. Another implementation of this protocol is available since
2013 with IBM z/OS Version 2 Release 1. 

SMC-R provides a “sockets over RDMA” solution that leverages industry
standard RDMA over Converged Ethernet (RoCE) technology.

IBM has developed a Linux implementation of the SMC-R standard. A new
socket protocol family AF_SMC is introduced. A preload library can be used
to enable TCP-based applications to use SMC-R without changes. 

Key aspects of SMC-R are: 
1. Provides optimized performance compared to standard TCP/IP over Ethernet
   within the data center for both request/response (latency) and streaming
   workloads (CPU savings) [3]. 
   Initial benchmarks on Linux on x86 processors have shown latency
   reduction of up to 52% with a throughput gain of 111% using SMC-R vs TCP
   for request/response message patterns (10 concurrent TCP connections
   with 16KB    messages) and CPU savings of up to 69% for streaming data
   patterns (single TCP connection with 20MB of data in one direction).
   [1] is currently updated to contain more detailed information on Linux
   and performance.
2. In order to preserve the traditional network administrative model the
   SMC-R protocol ties into the existing IP addresses and uses TCP's
   handshake to establish connections. This allows existing management
   tools and security infrastructure to control the creation of SMC
   connections.
3. The SMC-R protocol logically bonds multiple RoCE adapters together
   providing    redundancy with transparent fail-over for improved high
   availability, increased bandwidth and load balancing across multiple
   RDMA-capable devices.
4. Due to its handshake protocol, SMC-R is compatible with (transparent to)
   existing TCP connection load balancers that are commonly used in the
   enterprise data center environment for multi-tier application workloads.
5. SMC-R's handshake protocol allows for transparent fallback to TCP/IP,
   should one of the peers not be capable of the protocol.

Additional SMC-R overview and reference materials are available [1].  

The SMC-R “rendezvous" protocol eliminates the need for RDMA-CM and the
exchange occurs through an initial TCP connection. Building on a TCP
connection to establish an SMC-R connection solves many key requirements,
including #4 and #5 above.
The rendezvous process occurs in 2 phases: 
1. TCP/IP 3-way exchange:
   Initiated when both client and server indicate SMC-R capability by
   including TCP experimental options on the TCP/IP 3-way handshake (syn
   flows) as described in RFC6994 [4]. The ExID assigned by IANA is
   0xE2D4C3D9 [5]. 
2. SMC-R 3-way exchange:
   When both partners indicate SMC-R capability then at the completion of
   the 3-way TCP handshake the SMC-R layers in each peer take control of
   the TCP connection and exchange their RDMA credentials. If this 3-way
   exchange    completes successfully the connection continues using SMC-R.
   If the exchange is not successful the connections falls back to standard
   TCP/IP. 

References:
[1] SMC-R Overview and Reference Materials:
    http://www-01.ibm.com/software/network/commserver/SMCR/ 
[2] SMC-R Informational RFC:
    http://tools.ietf.org/html/draft-fox-tcpm-shared-memory-rdma-07
[3] Linux SMC-R Overview and Performance Summary
    (archs x86 and s390):
    http://www-01.ibm.com/software/network/commserver/SMCR/ 
[4] Shared Use of TCP Experimental Options RFC 6994:
    https://tools.ietf.org/rfc/rfc6994.txt    
[5] IANA ExID SMCR: 
    http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xhtml#tcp-exids

The patch series is prepared to apply to net-next and consists of these
parts:
1. include/.../tcp: TCP experimental option - definitions
2. net/ipv4/tcp:    TCP experimental option - TCP hooks for SMC
3. net:             definitions to establish new socket family
4. net/smc:         new socket family
5. net/smc:         static key enabling in smc

In the future, SMC-R will be enhanced to cover:
- IPv6 support
- Tracing
- Statistics support

shortlog:

Ursula Braun (5):
  tcp: TCP experimental option for SMC - definitions
  tcp: TCP experimental option for SMC - TCP hooks
  net: introduce socket family constants
  smc: introduce socket family AF_SMC
  smc: increase / decrease static key

 include/linux/socket.h   |    4 +-
 include/linux/tcp.h      |   14 +-
 include/net/inet_sock.h  |    3 +-
 include/net/smc.h        |   13 +
 include/net/tcp.h        |  136 ++
 net/Kconfig              |    1 +
 net/Makefile             |    1 +
 net/core/sock.c          |   20 +-
 net/ipv4/tcp.c           |    3 +
 net/ipv4/tcp_input.c     |    7 +
 net/ipv4/tcp_minisocks.c |    3 +
 net/ipv4/tcp_output.c    |   23 +-
 net/ipv4/tcp_timer.c     |    2 +-
 net/smc/Kconfig          |    9 +
 net/smc/Makefile         |    3 +
 net/smc/af_smc.c         | 3172 ++++++++++++++++++++++++++++++++++++++++++++
 net/smc/af_smc.h         |  706 ++++++++++
 net/smc/smc_core.c       | 3297 ++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_llc.c        | 1597 ++++++++++++++++++++++
 net/smc/smc_llc.h        |  192 +++
 net/smc/smc_proc.c       |  953 ++++++++++++++
 21 files changed, 10135 insertions(+), 24 deletions(-)
 create mode 100644 include/net/smc.h
 create mode 100644 net/smc/Kconfig
 create mode 100644 net/smc/Makefile
 create mode 100644 net/smc/af_smc.c
 create mode 100644 net/smc/af_smc.h
 create mode 100644 net/smc/smc_core.c
 create mode 100644 net/smc/smc_llc.c
 create mode 100644 net/smc/smc_llc.h
 create mode 100644 net/smc/smc_proc.c

-- 
2.3.8

[PATCH V3 net-next 1/5] tcp: TCP experimental option for SMC - definitions

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

The SMC-R protocol defines dynamic discovery of peers. This is done by
implementing experimental TCP options as defined in RFC6994. The TCP code
needs to be extended to support RFC6994.

Setting the TCP experimental option for SMC-R [2] will be triggered from
kernel exploiters like the new SMC-R socket family by setting a new
flag "syn_smc" on struct tcp_sock of the connecting and the listening
socket. If the client peer is SMC-R capable, flag syn_smc is kept on the
connecting socket after the 3-way TCP handshake, otherwise it is reset.
If the server peer is SMC-R capable, the new connected TCP socket has
the new flag set, otherwise not.

Code snippet client:
  tcp_sk(sock->sk)->syn_smc = 1;
  rc = kernel_connect(sock, addr, alen, flags);
  if (tcp_sk(sock->sk)->syn_smc) {
          /* switch to smc for this connection */

Code snippet server:
  tcp_sk(sock->sk)->syn_smc = 1;
  rc = kernel_listen(sock, backlog);
  rc = kernel_accept(sock, &newsock, 0);
  if (tcp_sk(newsock->sk)->syn_smc) {
          /* switch to smc for this connection */

References:
[1] Shared Use of TCP Experimental Options RFC 6994:
    https://tools.ietf.org/rfc/rfc6994.txt
[2] IANA ExID SMCR:
    http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xhtml#tcp-exids

This patch has already been posted in June 2013, but Dave Miller has
postponed applying till the user of the new flags, ie. the entire SMC-R
protocol stack is implemented.

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 include/linux/tcp.h     | 6 ++++--
 include/net/inet_sock.h | 3 ++-
 include/net/tcp.h       | 8 ++++++++
 3 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 48c3696..488a875 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -87,7 +87,8 @@ struct tcp_options_received {
 		tstamp_ok : 1,	/* TIMESTAMP seen on SYN packet		*/
 		dsack : 1,	/* D-SACK is scheduled			*/
 		wscale_ok : 1,	/* Wscale seen on SYN packet		*/
-		sack_ok : 4,	/* SACK seen on SYN packet		*/
+		sack_ok : 3,	/* SACK seen on SYN packet		*/
+		smc_ok : 1,	/* SMC seen on SYN packet		*/
 		snd_wscale : 4,	/* Window scaling received from sender	*/
 		rcv_wscale : 4;	/* Window scaling to send to receiver	*/
 	u8	num_sacks;	/* Number of SACK blocks		*/
@@ -207,7 +208,8 @@ struct tcp_sock {
 		syn_fastopen_exp:1,/* SYN includes Fast Open exp. option */
 		syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
 		save_syn:1,	/* Save headers of SYN packet */
-		is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
+		is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */
+		syn_smc:1;	/* SYN includes SMC			*/
 	u32	tlp_high_seq;	/* snd_nxt at the time of TLP retransmit. */
 
 /* RTT measurement */
diff --git a/include/net/inet_sock.h b/include/net/inet_sock.h
index 47eb67b..18d8d3f 100644
--- a/include/net/inet_sock.h
+++ b/include/net/inet_sock.h
@@ -91,7 +91,8 @@ struct inet_request_sock {
 				wscale_ok  : 1,
 				ecn_ok	   : 1,
 				acked	   : 1,
-				no_srccheck: 1;
+				no_srccheck: 1,
+				smc_ok     : 1;
 	kmemcheck_bitfield_end(flags);
 	u32                     ir_mark;
 	union {
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 364426a..e4584ed 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -46,6 +46,7 @@
 
 #include <linux/seq_file.h>
 #include <linux/memcontrol.h>
+#include <linux/unaligned/access_ok.h>
 
 extern struct inet_hashinfo tcp_hashinfo;
 
@@ -185,6 +186,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
  * experimental options. See draft-ietf-tcpm-experimental-options-00.txt
  */
 #define TCPOPT_FASTOPEN_MAGIC	0xF989
+#define TCPOPT_SMC_MAGIC	0xE2D4C3D9
 
 /*
  *     TCP option lengths
@@ -197,6 +199,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_MD5SIG         18
 #define TCPOLEN_FASTOPEN_BASE  2
 #define TCPOLEN_EXP_FASTOPEN_BASE  4
+#define TCPOLEN_EXP_SMC_BASE   6
 
 /* But this is what stacks really send out. */
 #define TCPOLEN_TSTAMP_ALIGNED		12
@@ -207,6 +210,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_SACK_PERBLOCK		8
 #define TCPOLEN_MD5SIG_ALIGNED		20
 #define TCPOLEN_MSS_ALIGNED		4
+#define TCPOLEN_EXP_SMC_BASE_ALIGNED    8
 
 /* Flags in tp->nonagle */
 #define TCP_NAGLE_OFF		1	/* Nagle's algo is disabled */
@@ -1762,4 +1766,8 @@ static inline void skb_set_tcp_pure_ack(struct sk_buff *skb)
 	skb->truesize = 2;
 }
 
+#if IS_ENABLED(CONFIG_AFSMC)
+extern struct static_key tcp_have_smc;
+#endif
+
 #endif	/* _TCP_H */
-- 
2.3.8

[PATCH V3 net-next 2/5] tcp: TCP experimental option for SMC - TCP hooks

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

The SMC-R protocol defines dynamic discovery of peers. This is done by
implementing experimental TCP options as defined in RFC6994. The TCP code
needs to be extended to support RFC6994.

Setting the TCP experimental option for SMC-R [2] will be triggered from
kernel exploiters like the new SMC-R socket family by setting a new
flag "syn_smc" on struct tcp_sock of the connecting and the listening
socket. If the client peer is SMC-R capable, flag syn_smc is kept on the
connecting socket after the 3-way TCP handshake, otherwise it is reset.
If the server peer is SMC-R capable, the new connected TCP socket has
the new flag set, otherwise not.

Code snippet client:
  tcp_sk(sock->sk)->syn_smc = 1;
  rc = kernel_connect(sock, addr, alen, flags);
  if (tcp_sk(sock->sk)->syn_smc) {
          /* switch to smc for this connection */

Code snippet server:
  tcp_sk(sock->sk)->syn_smc = 1;
  rc = kernel_listen(sock, backlog);
  rc = kernel_accept(sock, &newsock, 0);
  if (tcp_sk(newsock->sk)->syn_smc) {
          /* switch to smc for this connection */

References:
[1] Shared Use of TCP Experimental Options RFC 6994:
    https://tools.ietf.org/rfc/rfc6994.txt
[2] IANA ExID SMCR:
    http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xhtml#tcp-exids

This patch has already been posted in June 2013, but Dave Miller has
postponed applying till the user of the new flags, ie. the entire SMC-R
protocol stack is implemented.

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 include/linux/tcp.h      |   8 +++
 include/net/tcp.h        | 128 +++++++++++++++++++++++++++++++++++++++++++++++
 net/ipv4/tcp.c           |   3 ++
 net/ipv4/tcp_input.c     |   7 +++
 net/ipv4/tcp_minisocks.c |   3 ++
 net/ipv4/tcp_output.c    |  23 +++------
 6 files changed, 155 insertions(+), 17 deletions(-)

diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 488a875..4afaa202 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -96,10 +96,18 @@ struct tcp_options_received {
 	u16	mss_clamp;	/* Maximal mss, negotiated at connection setup */
 };
 
+#if IS_ENABLED(CONFIG_AFSMC)
+extern struct static_key tcp_have_smc;
+#endif
+
 static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 {
 	rx_opt->tstamp_ok = rx_opt->sack_ok = 0;
 	rx_opt->wscale_ok = rx_opt->snd_wscale = 0;
+#if IS_ENABLED(CONFIG_AFSMC)
+	if (static_key_false(&tcp_have_smc))
+		rx_opt->smc_ok = 0;
+#endif
 }
 
 /* This is the max number of SACKS that we'll generate and process. It's safe
diff --git a/include/net/tcp.h b/include/net/tcp.h
index e4584ed..4e28233 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -1766,8 +1766,136 @@ static inline void skb_set_tcp_pure_ack(struct sk_buff *skb)
 	skb->truesize = 2;
 }
 
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
 #if IS_ENABLED(CONFIG_AFSMC)
 extern struct static_key tcp_have_smc;
 #endif
 
+static inline void smc_parse_options(const struct tcphdr *th,
+				     struct tcp_options_received *opt_rx,
+				     const unsigned char *ptr,
+				     int opsize)
+{
+#if IS_ENABLED(CONFIG_AFSMC)
+	if (!static_key_false(&tcp_have_smc))
+		return;
+	if (th->syn && !(opsize & 1) &&
+	    opsize >= TCPOLEN_EXP_SMC_BASE &&
+	    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC)
+		opt_rx->smc_ok = 1;
+#endif
+}
+
+static inline void smc_options_write(__be32 *ptr, u16 *options)
+{
+#if IS_ENABLED(CONFIG_AFSMC)
+	if (!static_key_false(&tcp_have_smc))
+		return;
+	if (unlikely(OPTION_SMC & *options)) {
+		*ptr++ = htonl((TCPOPT_NOP  << 24) |
+			       (TCPOPT_NOP  << 16) |
+			       (TCPOPT_EXP <<  8) |
+			       (TCPOLEN_EXP_SMC_BASE));
+		*ptr++ = htonl(TCPOPT_SMC_MAGIC);
+	}
+#endif
+}
+
+static inline void smc_set_option(struct tcp_sock *tp,
+				  struct tcp_out_options *opts,
+				  unsigned int *remaining)
+{
+#if IS_ENABLED(CONFIG_AFSMC)
+	if (!static_key_false(&tcp_have_smc))
+		return;
+	if (tp->syn_smc) {
+		u32 need = TCPOLEN_EXP_SMC_BASE_ALIGNED;
+
+		if (*remaining >= need) {
+			opts->options |= OPTION_SMC;
+			*remaining -= need;
+		}
+	}
+#endif
+
+}
+static inline void smc_set_option_cond(struct tcp_sock *tp,
+				       struct request_sock *req,
+				       struct tcp_out_options *opts,
+				       unsigned int *remaining)
+{
+#if IS_ENABLED(CONFIG_AFSMC)
+	struct inet_request_sock *ireq;
+
+	if (!static_key_false(&tcp_have_smc))
+		return;
+
+	ireq = inet_rsk(req);
+	if (tp->syn_smc && ireq->smc_ok) {
+		u32 need = TCPOLEN_EXP_SMC_BASE_ALIGNED;
+
+		if (*remaining >= need) {
+			opts->options |= OPTION_SMC;
+			*remaining -= need;
+		}
+	}
+#endif
+}
+
+static inline void smc_set_capability(struct inet_request_sock *ireq,
+				      const struct tcp_options_received *rx_opt)
+{
+#if IS_ENABLED(CONFIG_AFSMC)
+	if (!static_key_false(&tcp_have_smc))
+		return;
+
+	if (rx_opt->smc_ok)
+		ireq->smc_ok = 1;
+	else
+		ireq->smc_ok = 0;
+#endif
+}
+
+static inline void smc_check_reset_syn(struct tcp_sock *tp)
+{
+#if IS_ENABLED(CONFIG_AFSMC)
+	if (static_key_false(&tcp_have_smc))
+		if (tp->syn_smc && !tp->rx_opt.smc_ok)
+			tp->syn_smc = 0;
+#endif
+}
+
+static inline void smc_check_reset_syn_req(struct tcp_sock *oldtp,
+					   struct request_sock *req,
+					   struct tcp_sock *newtp)
+{
+#if IS_ENABLED(CONFIG_AFSMC)
+	struct inet_request_sock *ireq;
+
+	if (!static_key_false(&tcp_have_smc))
+		return;
+
+	ireq = inet_rsk(req);
+	if (oldtp->syn_smc && !ireq->smc_ok)
+		newtp->syn_smc = 0;
+#endif
+}
+
 #endif	/* _TCP_H */
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 7f40567..4d27db6 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -301,6 +301,9 @@ EXPORT_SYMBOL(sysctl_tcp_wmem);
 atomic_long_t tcp_memory_allocated;	/* Current allocated memory. */
 EXPORT_SYMBOL(tcp_memory_allocated);
 
+struct static_key tcp_have_smc __read_mostly = STATIC_KEY_INIT_FALSE;
+EXPORT_SYMBOL(tcp_have_smc);
+
 /*
  * Current number of TCP sockets.
  */
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 1578fc2..4d35ceb 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -3748,6 +3748,9 @@ void tcp_parse_options(const struct sk_buff *skb,
 					tcp_parse_fastopen_option(opsize -
 						TCPOLEN_EXP_FASTOPEN_BASE,
 						ptr + 2, th->syn, foc, true);
+				else
+					smc_parse_options(th, opt_rx, ptr,
+							  opsize);
 				break;
 
 			}
@@ -5556,6 +5559,8 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		 * is initialized. */
 		tp->copied_seq = tp->rcv_nxt;
 
+		smc_check_reset_syn(tp);
+
 		smp_mb();
 
 		tcp_finish_connect(sk, skb);
@@ -6017,6 +6022,8 @@ static void tcp_openreq_init(struct request_sock *req,
 	ireq->ir_rmt_port = tcp_hdr(skb)->source;
 	ireq->ir_num = ntohs(tcp_hdr(skb)->dest);
 	ireq->ir_mark = inet_request_mark(sk, skb);
+	smc_set_capability(ireq, rx_opt);
+
 }
 
 struct request_sock *inet_reqsk_alloc(const struct request_sock_ops *ops,
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 6d8795b..62e6c2c 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -443,6 +443,9 @@ struct sock *tcp_create_openreq_child(struct sock *sk, struct request_sock *req,
 		struct tcp_request_sock *treq = tcp_rsk(req);
 		struct inet_connection_sock *newicsk = inet_csk(newsk);
 		struct tcp_sock *newtp = tcp_sk(newsk);
+		struct tcp_sock *oldtp = tcp_sk(sk);
+
+		smc_check_reset_syn_req(oldtp, req, newtp);
 
 		/* Now setup tcp_sock */
 		newtp->pred_flags = 0;
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 7105784..17ddabd 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -419,23 +419,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
  *
  * Beware: Something in the Internet is very sensitive to the ordering of
@@ -542,6 +525,8 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 		}
 		ptr += (len + 3) >> 2;
 	}
+
+	smc_options_write(ptr, &options);
 }
 
 /* Compute TCP options for SYN packets. This is not the final
@@ -609,6 +594,8 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	smc_set_option(tp, opts, &remaining);
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -670,6 +657,8 @@ static unsigned int tcp_synack_options(struct sock *sk,
 		}
 	}
 
+	smc_set_option_cond(tcp_sk(sk), req, opts, &remaining);
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
-- 
2.3.8

[PATCH V3 net-next 3/5] net: introduce socket family constants

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

The new socket family is assigned the next available address / protocol
family constant 41.
Implementing SO_KEEPALIVE for SMC-R requires an extra hook in net/ipv4/timer.c.

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 include/linux/socket.h |  4 +++-
 include/net/smc.h      | 13 +++++++++++++
 net/ipv4/tcp_timer.c   |  2 +-
 3 files changed, 17 insertions(+), 2 deletions(-)
 create mode 100644 include/net/smc.h

diff --git a/include/linux/socket.h b/include/linux/socket.h
index 5bf59c8..dd18d32 100644
--- a/include/linux/socket.h
+++ b/include/linux/socket.h
@@ -200,7 +200,8 @@ struct ucred {
 #define AF_ALG		38	/* Algorithm sockets		*/
 #define AF_NFC		39	/* NFC sockets			*/
 #define AF_VSOCK	40	/* vSockets			*/
-#define AF_MAX		41	/* For now.. */
+#define AF_SMC		41	/* smc sockets>->------->-------*/
+#define AF_MAX		42	/* For now.. */
 
 /* Protocol families, same as address families. */
 #define PF_UNSPEC	AF_UNSPEC
@@ -246,6 +247,7 @@ struct ucred {
 #define PF_ALG		AF_ALG
 #define PF_NFC		AF_NFC
 #define PF_VSOCK	AF_VSOCK
+#define PF_SMC		AF_SMC
 #define PF_MAX		AF_MAX
 
 /* Maximum queue length specifiable by listen.  */
diff --git a/include/net/smc.h b/include/net/smc.h
new file mode 100644
index 0000000..cd513ee
--- /dev/null
+++ b/include/net/smc.h
@@ -0,0 +1,13 @@
+/*
+ * SMC		Definitions for the SMC protocol.
+ *
+ * Author:	Ursula Braun <ursula.braun@de.ibm.com>
+ */
+#ifndef _SMC_H
+#define _SMC_H
+
+/* SMC socket options - disjunct with TCP socket options */
+#define SMC_KEEPALIVE		99	/* start/stop keepalives */
+
+#endif /* _SMC_H */
+
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 7149ebc..070bfc7 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -557,7 +557,7 @@ void tcp_set_keepalive(struct sock *sk, int val)
 	else if (!val)
 		inet_csk_delete_keepalive_timer(sk);
 }
-
+EXPORT_SYMBOL(tcp_set_keepalive);
 
 static void tcp_keepalive_timer (unsigned long data)
 {
-- 
2.3.8

[PATCH V3 net-next 4/5] smc: introduce socket family AF_SMC

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

In 2013, IBM introduced an optimized communications solution for the
IBM zEnterprise EC12 and BC12 (s390 in Linux terminology) that is
comprised of the IBM 10GbE RoCE Express feature with Shared Memory
Communications-RDMA (SMC-R) protocol [1].
SMC-R is designed for the enterprise data center environment and is an open
protocol as specified in the informational RFC [2]. The final draft
submitted by IBM has been approved for publication and is in the final
editorial stage. Another implementation of this protocol is available since
2013 with IBM z/OS Version 2 Release 1.

SMC-R provides a “sockets over RDMA” solution that leverages industry
standard RDMA over Converged Ethernet (RoCE) technology.

IBM has developed a Linux implementation of the SMC-R standard. A new
socket protocol family AF_SMC is introduced. A preload library can be used
to enable TCP-based applications to use SMC-R without changes.

Key aspects of SMC-R are:
1. Provides optimized performance compared to standard TCP/IP over Ethernet
   within the data center for both request/response (latency) and streaming
   workloads (CPU savings) [3].
2. In order to preserve the traditional network administrative model the
   SMC-R protocol ties into the existing IP addresses and uses TCP's
   handshake to establish connections. This allows existing management
   tools and security infrastructure to control the creation of SMC
   connections.
3. The SMC-R protocol logically bonds multiple RoCE adapters together
   providing redundancy with transparent fail-over for improved high
   availability, increased bandwidth and load balancing across multiple
   RDMA-capable devices.
4. Due to its handshake protocol, SMC-R is compatible with (transparent to)
   existing TCP connection load balancers that are commonly used in the
   enterprise data center environment for multi-tier application workloads.
5. SMC-R's handshake protocol allows for transparent fallback to TCP/IP,
   should one of the peers not be capable of the protocol.

Additional SMC-R overview and reference materials are available [1].

The SMC-R “rendezvous" protocol eliminates the need for RDMA-CM and the
exchange occurs through an initial TCP connection. Building on a TCP
connection to establish an SMC-R connection solves many key requirements,
including #4 and #5 above.
The rendezvous process occurs in 2 phases:
1. TCP/IP 3-way exchange:
   Initiated when both client and server indicate SMC-R capability by
   including TCP experimental options on the TCP/IP 3-way handshake (syn
   flows) as    described in RFC6994 [4]. The ExID assigned by IANA is
   0xE2D4C3D9 [5].
2. SMC-R 3-way exchange:
   When both partners indicate SMC-R capability then at the completion of
   the 3-way TCP handshake the SMC-R layers in each peer take control of
   the TCP connection and exchange their RDMA credentials. If this 3-way
   exchange completes successfully the connection continues using SMC-R.
   If the exchange is not successful the connections falls back to standard
   TCP/IP.

A connection using SMC-R starts out setting up a connection using the
TCP/IP protocol [2]. Thus an internal smc-managed TCP socket is
established, whose traffic flows across an interface belonging to the same
Converged Ethernet fabric, defined by a so-called "pnet table" [1].

As part of the connection setup, flags indicating the support
for SMC-R are used to negotiate SMC-R usage. If any of the two parties
involved does not support SMC-R, the connection falls back to plain TCP/IP
usage (the previously established TCP connection will be used for any
further traffic). If both parties support SMC-R, the TCP/IP connection is
used to exchange the necessary connection data for setup of RoCE RC-QP
connection in the so-called "rendezvous handshake".

The code also implements also the link management of the SMC-R protocol
forming link groups for each remote host and provide for
- dynamic addition of further RoCE Host Channel Adapters (HCAs) to form a
  so-called link group
- seamless failover, moving existing connections from a failed HCA
  to the remaining HCAs in a link group
- load balancing of connections across the links in a link group

SMC-R design defines memory areas for data transport through RDMA called
RDMA Memory Buffer (RMB) which are split up into several RMB Elements
(RMBEs) assigned to single connections. This initial Linux implementation
requires real contiguous storage for these RMBs and uses a separate RMB
with just 1 RMBE for every connection. Its size is derived from:
- a system wide default minimum SMC receive buffer size (tailorable by
  /proc/net/smc/rcvbuf
- if given, the socket specific SETSOCKOPT SO_RCVBUF configuration
- the size of the tcp socket receive buffer determined from the system wide
  tcp socket receive buffer configuration
- the amount of available contiguous storage
In the Linux implementation, an equivalent concept exists for the socket
send buffers [1].

In the future, SMC-R will be enhanced to cover:
- IPv6 support
- Tracing
- Statistic hooks

References:
[1] SMC-R Overview and Reference Materials:
    http://www-01.ibm.com/software/network/commserver/SMCR/
[2] SMC-R Informational RFC:
    http://tools.ietf.org/html/draft-fox-tcpm-shared-memory-rdma-07
[3] Linux SMC-R Overview and Performance Summary
    (archs x86 and s390):
    http://www-01.ibm.com/software/network/commserver/SMCR/
[4] Shared Use of TCP Experimental Options RFC 6994:
    https://tools.ietf.org/rfc/rfc6994.txt
[5] IANA ExID SMCR:
    http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xhtml#tcp-exids

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 net/Kconfig        |    1 +
 net/Makefile       |    1 +
 net/core/sock.c    |   20 +-
 net/smc/Kconfig    |    9 +
 net/smc/Makefile   |    3 +
 net/smc/af_smc.c   | 3170 ++++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/af_smc.h   |  706 +++++++++++
 net/smc/smc_core.c | 3297 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_llc.c  | 1597 +++++++++++++++++++++++++
 net/smc/smc_llc.h  |  192 +++
 net/smc/smc_proc.c |  953 +++++++++++++++
 11 files changed, 9947 insertions(+), 2 deletions(-)
 create mode 100644 net/smc/Kconfig
 create mode 100644 net/smc/Makefile
 create mode 100644 net/smc/af_smc.c
 create mode 100644 net/smc/af_smc.h
 create mode 100644 net/smc/smc_core.c
 create mode 100644 net/smc/smc_llc.c
 create mode 100644 net/smc/smc_llc.h
 create mode 100644 net/smc/smc_proc.c

diff --git a/net/Kconfig b/net/Kconfig
index 7021c1b..57435aa 100644
--- a/net/Kconfig
+++ b/net/Kconfig
@@ -54,6 +54,7 @@ source "net/packet/Kconfig"
 source "net/unix/Kconfig"
 source "net/xfrm/Kconfig"
 source "net/iucv/Kconfig"
+source "net/smc/Kconfig"
 
 config INET
 	bool "TCP/IP networking"
diff --git a/net/Makefile b/net/Makefile
index 3995613..33a5f54 100644
--- a/net/Makefile
+++ b/net/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_MAC80211)		+= mac80211/
 obj-$(CONFIG_TIPC)		+= tipc/
 obj-$(CONFIG_NETLABEL)		+= netlabel/
 obj-$(CONFIG_IUCV)		+= iucv/
+obj-$(CONFIG_AFSMC)		+= smc/
 obj-$(CONFIG_RFKILL)		+= rfkill/
 obj-$(CONFIG_NET_9P)		+= 9p/
 obj-$(CONFIG_CAIF)		+= caif/
diff --git a/net/core/sock.c b/net/core/sock.c
index 08f16db..4a10166 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -139,6 +139,9 @@
 
 #ifdef CONFIG_INET
 #include <net/tcp.h>
+#if IS_ENABLED(CONFIG_AFSMC)
+#include <net/smc.h>
+#endif /* CONFIG_AFSMC */
 #endif
 
 #include <net/busy_poll.h>
@@ -784,8 +787,21 @@ set_rcvbuf:
 	case SO_KEEPALIVE:
 #ifdef CONFIG_INET
 		if (sk->sk_protocol == IPPROTO_TCP &&
-		    sk->sk_type == SOCK_STREAM)
-			tcp_set_keepalive(sk, valbool);
+		    sk->sk_type == SOCK_STREAM) {
+#if IS_ENABLED(CONFIG_AFSMC)
+			if (sk->sk_family == AF_SMC) {
+				/* cannot use the TCP-way, since AFSMC
+				 * should be allowed to be a module
+				 */
+				release_sock(sk);
+				sock->ops->setsockopt(sock, SOL_TCP,
+						      SMC_KEEPALIVE,
+						      optval, optlen);
+				lock_sock(sk);
+			} else
+#endif
+				tcp_set_keepalive(sk, valbool);
+		}
 #endif
 		sock_valbool_flag(sk, SOCK_KEEPOPEN, valbool);
 		break;
diff --git a/net/smc/Kconfig b/net/smc/Kconfig
new file mode 100644
index 0000000..4ae42d5
--- /dev/null
+++ b/net/smc/Kconfig
@@ -0,0 +1,9 @@
+config AFSMC
+	tristate "AF_SMC socket address family"
+	depends on INET && INFINIBAND
+	---help---
+	  SMC_R provides a "sockets over RDMA" solution making use of
+	  RDMA over Converged Ethernet (ROCE) technology.
+	  Socket family AF_SMC implements this solution for Linux.
+
+	  Select this option if you want to run AF_SMC socket applications
diff --git a/net/smc/Makefile b/net/smc/Makefile
new file mode 100644
index 0000000..76017f0
--- /dev/null
+++ b/net/smc/Makefile
@@ -0,0 +1,3 @@
+ccflags-y += -I$(src)
+smc-y := af_smc.o smc_core.o smc_proc.o smc_llc.o
+obj-$(CONFIG_AFSMC)	+= smc.o
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
new file mode 100644
index 0000000..ed62370
--- /dev/null
+++ b/net/smc/af_smc.c
@@ -0,0 +1,3170 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  AF_SMC protocol family socket handler
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              Stefan Raspl <raspl@linux.vnet.ibm.com>
+ *              based on prototype from Frank Blaschka
+ */
+
+#define KMSG_COMPONENT "af_smc"
+#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/seq_file.h>
+#include <linux/uaccess.h>
+#include <linux/workqueue.h>
+#include <linux/proc_fs.h>
+#include <linux/types.h>
+#include <net/sock.h>
+#include <net/inet_sock.h>
+#include <net/tcp.h>
+#include <net/smc.h>
+#include <asm/byteorder.h>
+#include <asm/ioctls.h>
+#include <rdma/ib_verbs.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+#define VERSION "1.0"
+
+static struct proto smc_proto = {
+	.name		= "AF_SMC",
+	.owner		= THIS_MODULE,
+	.obj_size	= sizeof(struct smc_sock),
+};
+
+static struct sock *smc_sock_alloc(struct socket *, int, gfp_t);
+static void smc_tcp_listen_worker(struct work_struct *);
+static int smc_getsockopt_from_tcp(struct smc_sock *, int);
+static struct sock *smc_accept_dequeue(struct sock *, struct socket *);
+
+struct list_head smc_ib_devices;	/* list of smc_ib_devices */
+static spinlock_t smc_ib_dev_lock;	/* protect smc_ib_devices_list */
+struct smc_lgr_list_struct smc_lgr_list;	/* list of link_groups */
+struct list_head smc_lgr_freeing_list;	/* list of link groups to be freed */
+static u8 local_peerid[8] = SMC_LOCAL_PEERID_RESET;	/* unique system
+							 * identifier
+							 */
+u32 smc_ctrl_buf_cnt = 16;		/* # of control buffers per link */
+u32 smc_max_conn_per_lgr = 200;		/* max. # of connections per lgr */
+u32 smc_port_add_delay;		/* port add delay (for certain switches) */
+atomic_t smc_lgr_num = ATOMIC_INIT(0);	/* unique link group number */
+unsigned int smc_def_rcvbuf_size = 65532;	/* def. receive buffer size */
+unsigned int smc_def_sndbuf_size = 65532;	/* def. send buffer size */
+
+atomic_t smc_reconfiguring = ATOMIC_INIT(0);	/* serialize port_add/port_err
+						 * and free_lgr
+						 */
+
+#ifdef CONFIG_S390
+#include <asm/ebcdic.h>
+#endif
+
+/* link is up - establish alternate link if applicable
+ * must be called with lgr->conf_mutex taken
+ */
+void smc_link_up(struct smc_link_group *lgr)
+{
+	struct smc_link *link = NULL;
+	struct smc_roce_defs rocdefs;
+	int i;
+	u8 p1, p2;
+
+	/* determine working link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			link = &lgr->lnk[i];
+			break;
+		}
+	}
+	if (!link || (lgr->lgr_type == SYMMETRIC) ||
+	    (lgr->lgr_type == ASYMMETRIC_PEER))
+		return;
+again:
+	if (!lgr->port_up)
+		return;
+	p1 = lgr->port_up & 1;
+	p2 = lgr->port_up & 2;
+	if (lgr->role == SMC_SERV) {
+		struct llc_add_link_fake_msg add_llc;
+
+		memset(&add_llc, 0, sizeof(add_llc));
+		add_llc.hd.length = sizeof(struct llc_msg);
+		add_llc.hd.type = LLC_ADD_LINK;
+		add_llc.ibdev = lgr->ibdev_up;
+		add_llc.port_num = p1 ? p1 : p2;
+		add_llc.hd.flags |= LLC_FLAG_LOCAL;
+		llc_enqueue(link, (struct llc_msg *)&add_llc);
+		lgr->port_up &= ~add_llc.port_num;
+	} else {
+		/* invite server to start ADD LINK */
+		rocdefs.ibdev = lgr->ibdev_up;
+		rocdefs.port = p1 ? p1 : p2;
+		rocdefs.vlan = lgr->vlan;
+		smc_gid_by_dev(&rocdefs);
+		if (lgr->llc_ctl[LLC_SERV].active != LLC_GRP_NONE) {
+			mutex_unlock(&lgr->conf_mutex);
+			wait_event_interruptible_timeout(lgr->llc_waiter,
+				(lgr->llc_ctl[LLC_SERV].active == LLC_GRP_NONE),
+				LLC_WAIT_TIMEO);
+			mutex_lock(&lgr->conf_mutex);
+		}
+		if (!list_empty(&lgr->list)) {
+			if (smc_port_active(rocdefs.ibdev, rocdefs.port))
+				llc_send_add_link(link, &rocdefs, NULL, LLC_REQ,
+						  LLC_NO_NOTIFY);
+		}
+		lgr->port_up &= ~rocdefs.port;
+	}
+	goto again;
+}
+
+static void smc_port_add_worker(struct work_struct *work)
+{
+	struct smc_ib_device *smc_ibdev =
+				container_of(work, struct smc_ib_device,
+					     port_add_work.work);
+	struct smc_link_group *lgr;
+	u8 port_num = 0;
+	int i;
+
+	atomic_inc(&smc_reconfiguring);
+	for (i = 0; i < 2; i++) {
+		char pnet[SMC_MAX_PNET_ID_LEN];
+		int rc;
+
+		if (!smc_ibdev->port_add[i])
+			continue;
+		smc_check_port_attr(smc_ibdev, i);
+		if (smc_ibdev->pattr[i].state != IB_PORT_ACTIVE)
+			continue;
+		port_num = i + 1;
+		memset(pnet, 0, sizeof(pnet));
+		rc = smc_pnet_by_ibdev(port_num, pnet, smc_ibdev);
+		if (rc)
+			continue;
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+			if (strncmp(pnet, lgr->pnet_id, sizeof(pnet)) ||
+			    (lgr->lgr_type == SYMMETRIC) ||
+			    (lgr->lgr_type == ASYMMETRIC_PEER))
+				continue;
+			if (cmpxchg(&lgr->lnk_up, 0, 1) == 0) {
+				lgr->ibdev_up = smc_ibdev;
+				lgr->port_up |= port_num;
+				schedule_work(&lgr->link_up_work);
+			}
+		}
+		spin_unlock_bh(&smc_lgr_list.lock);
+		smc_ibdev->port_add[i] = 0;
+	}
+	atomic_dec(&smc_reconfiguring);
+}
+
+/* link is down - switch connections to alternate link
+ * must be called with lgr->conf_mutex taken
+ */
+void smc_link_down(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct smc_link *link = &lgr->lnk[lnk_idx];
+	struct smc_ib_device *smc_ibdev = link->roce_defs.ibdev;
+	int to_lnk_idx;
+	u8 port_num = link->roce_defs.port;
+
+	smc_modify_qp_reset(link);
+	to_lnk_idx = smc_switch_conns(lgr, lnk_idx, 1);
+	if (to_lnk_idx < 0) { /* no backup link available */
+		smc_free_link(lgr, lnk_idx);
+		return;
+	}
+	lgr->lgr_type = SINGLE;
+	if (lgr->role == SMC_SERV) {
+		struct llc_del_link_msg del_llc;
+
+		memset(&del_llc, 0, sizeof(del_llc));
+		del_llc.hd.length = sizeof(struct llc_msg);
+		del_llc.hd.type = LLC_DEL_LINK;
+		del_llc.link_num = link->link_id;
+		del_llc.reason = htonl(LLC_DEL_LOST_PATH);
+		del_llc.hd.flags |= LLC_FLAG_DEL_ORDERLY;
+		del_llc.hd.flags |= LLC_FLAG_LOCAL;
+		llc_enqueue(&lgr->lnk[to_lnk_idx], (struct llc_msg *)&del_llc);
+		/* try to restart the link */
+		if (!list_empty(&lgr->list) &&
+		    !list_empty(&smc_ibdev->list) &&
+		    smc_port_active(smc_ibdev, port_num)) {
+			lgr->ibdev_up = smc_ibdev;
+			lgr->port_up |= port_num;
+			schedule_work(&lgr->link_up_work);
+		}
+	} else {
+		if (lgr->llc_ctl[LLC_SERV].active != LLC_GRP_NONE) {
+			mutex_unlock(&lgr->conf_mutex);
+			wait_event_interruptible_timeout(lgr->llc_waiter,
+				(lgr->llc_ctl[LLC_SERV].active ==
+							LLC_GRP_NONE),
+				LLC_WAIT_TIMEO);
+			mutex_lock(&lgr->conf_mutex);
+		}
+		llc_send_del_link(&lgr->lnk[to_lnk_idx], link,
+				  LLC_FLAG_DEL_ORDERLY,
+				  LLC_DEL_LOST_PATH, LLC_NO_NOTIFY);
+	}
+}
+
+static void smc_port_err_worker(struct work_struct *work)
+{
+	struct smc_ib_device *smc_ibdev =
+		container_of(work, struct smc_ib_device, port_err_work);
+	struct smc_link_group *lgr;
+	int i, rc1, rc2;
+	u8 port_num;
+	char pnet1[SMC_MAX_PNET_ID_LEN], pnet2[SMC_MAX_PNET_ID_LEN];
+
+	atomic_inc(&smc_reconfiguring);
+	for (i = 0; i < 2; i++) {
+		if (!smc_ibdev->port_err[i])
+			continue;
+		smc_check_port_attr(smc_ibdev, i);
+		port_num = i + 1;
+		rc1 = smc_pnet_by_ibdev(1, pnet1, smc_ibdev);
+		rc2 = smc_pnet_by_ibdev(2, pnet2, smc_ibdev);
+		smc_ibdev->port_err[i] = 0;
+		if (rc1 && rc2) /* no pnet definition for this device */
+			continue;
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+			int j;
+
+			if (strncmp(pnet1, lgr->pnet_id, sizeof(pnet1)) &&
+			    strncmp(pnet2, lgr->pnet_id, sizeof(pnet2)))
+				continue;
+			if (lgr->lgr_type == NONE) /* no links */
+				continue;
+			for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+				struct smc_link *lnk = &lgr->lnk[j];
+
+				if ((lnk->roce_defs.ibdev == smc_ibdev) &&
+				    (lnk->roce_defs.port == port_num) &&
+				    cmpxchg(&lgr->lnk_down[j], 0, 1) == 0)
+					schedule_work(&lgr->link_down_work);
+			}
+		}
+		spin_unlock_bh(&smc_lgr_list.lock);
+	}
+	atomic_dec(&smc_reconfiguring);
+}
+
+static void smc_global_event_handler(struct ib_event_handler *handler,
+				     struct ib_event *event)
+{
+	struct smc_ib_device *dev;
+	u8 port_num = event->element.port_num - 1;
+
+	/* irq context! */
+	dev = container_of(handler, struct smc_ib_device, event_handler);
+	switch (event->event) {
+	case IB_EVENT_DEVICE_FATAL:
+		dev->port_err[0] = 1;
+		dev->port_err[1] = 1;
+		if ((cmpxchg(&dev->port_err[0], 0, 1) == 0) ||
+		    (cmpxchg(&dev->port_err[1], 0, 1) == 0))
+			schedule_work(&dev->port_err_work);
+		break;
+	case IB_EVENT_PORT_ERR:
+		if (cmpxchg(&dev->port_err[port_num], 0, 1) == 0)
+			schedule_work(&dev->port_err_work);
+		break;
+	case IB_EVENT_PORT_ACTIVE:
+		if (cmpxchg(&dev->port_add[port_num], 0, 1) == 0)
+			schedule_delayed_work(&dev->port_add_work,
+					      smc_port_add_delay);
+		break;
+	case IB_EVENT_GID_CHANGE:
+		break;
+	default:
+		break;
+	}
+}
+
+static void smc_ib_add_dev(struct ib_device *dev)
+{
+	struct smc_ib_device *smc_ibdev;
+	struct smc_roce_defs rocdefs;
+	struct ib_cq_init_attr attr = {.cqe = SMC_MAX_CQE, .comp_vector = 0};
+
+	if (dev->node_type != RDMA_NODE_IB_CA)
+		return;
+	smc_ibdev = kzalloc(sizeof(*smc_ibdev), GFP_KERNEL);
+	if (!smc_ibdev)
+		return;
+	smc_ibdev->dev = dev;
+	smc_ibdev->roce_cq_send = ib_create_cq(dev, smc_cq_handler_send,
+		smc_cq_event_handler, smc_ibdev, &attr);
+	if (IS_ERR(smc_ibdev->roce_cq_send))
+		goto err;
+	smc_ibdev->roce_cq_recv = ib_create_cq(dev, smc_cq_handler_recv,
+		smc_cq_event_handler, smc_ibdev, &attr);
+	if (IS_ERR(smc_ibdev->roce_cq_recv))
+		goto err;
+	if (!strncmp(local_peerid, SMC_LOCAL_PEERID_RESET,
+		     sizeof(local_peerid))) {
+		rocdefs.ibdev = smc_ibdev;
+		rocdefs.port = 1;
+		smc_get_ib_mac(&rocdefs);
+		memcpy(&local_peerid[2], &rocdefs.mac, sizeof(rocdefs.mac));
+		get_random_bytes_arch(&local_peerid[0], 2);
+	}
+	spin_lock_bh(&smc_ib_dev_lock);
+	list_add_tail(&smc_ibdev->list, &smc_ib_devices);
+	spin_unlock_bh(&smc_ib_dev_lock);
+	ib_set_client_data(dev, &smc_ib_client, smc_ibdev);
+	smc_check_dev_attr(smc_ibdev);
+	INIT_WORK(&smc_ibdev->port_err_work, smc_port_err_worker);
+	INIT_DELAYED_WORK(&smc_ibdev->port_add_work, smc_port_add_worker);
+	INIT_IB_EVENT_HANDLER(&smc_ibdev->event_handler, dev,
+			      smc_global_event_handler);
+	ib_register_event_handler(&smc_ibdev->event_handler);
+	tasklet_init(&smc_ibdev->recv_tasklet, smc_recv_tasklet_fn,
+		     (unsigned long)smc_ibdev);
+	tasklet_init(&smc_ibdev->send_tasklet, smc_send_tasklet_fn,
+		     (unsigned long)smc_ibdev);
+	return;
+
+err:
+	kfree(smc_ibdev);
+}
+
+static void smc_wait_for_freed_links(struct smc_ib_device *sdev)
+{
+	struct smc_link_group *lgr;
+	struct smc_link *lnk;
+	int i;
+
+again:
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			lnk = &lgr->lnk[i];
+			if (lnk->roce_defs.ibdev == sdev) {
+				spin_unlock_bh(&smc_lgr_list.lock);
+				switch (atomic_read(&lnk->state)) {
+				case SMC_LINK_FREED:
+				case SMC_LINK_UP: /* wait for port_err finish */
+					msleep(LLC_WAIT_FIRST_TIMEO);
+					break;
+				case SMC_LINK_DOWN:
+					smc_free_link(lgr, i);
+					break;
+				default:
+					break;
+				}
+				goto again;
+			}
+		}
+	}
+	list_for_each_entry(lgr, &smc_lgr_freeing_list, list) {
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			lnk = &lgr->lnk[i];
+			if (lnk->roce_defs.ibdev == sdev) {
+				spin_unlock_bh(&smc_lgr_list.lock);
+				switch (atomic_read(&lnk->state)) {
+				case SMC_LINK_FREED:
+					msleep(LLC_WAIT_FIRST_TIMEO);
+					break;
+				case SMC_LINK_UP:
+					atomic_set(&lnk->state, SMC_LINK_DOWN);
+					/* fall through */
+				case SMC_LINK_DOWN:
+					smc_free_link(lgr, i);
+					break;
+				default:
+					break;
+				}
+				goto again;
+			}
+		}
+	}
+	spin_unlock_bh(&smc_lgr_list.lock);
+}
+
+static void smc_ib_remove_dev(struct ib_device *dev)
+{
+	struct smc_ib_device *smc_ibdev;
+
+	atomic_inc(&smc_reconfiguring);
+	smc_ibdev = ib_get_client_data(dev, &smc_ib_client);
+	ib_unregister_event_handler(&smc_ibdev->event_handler);
+	smc_ibdev->port_add[0] = 0;
+	smc_ibdev->port_add[1] = 0;
+	smc_ibdev->port_err[0] = 1;
+	smc_ibdev->port_err[1] = 1;
+	schedule_work(&smc_ibdev->port_err_work);
+	ib_set_client_data(dev, &smc_ib_client, NULL);
+	spin_lock_bh(&smc_ib_dev_lock);
+	list_del_init(&smc_ibdev->list);
+	spin_unlock_bh(&smc_ib_dev_lock);
+	smc_wait_for_freed_links(smc_ibdev);
+	tasklet_kill(&smc_ibdev->recv_tasklet);
+	tasklet_kill(&smc_ibdev->send_tasklet);
+	ib_destroy_cq(smc_ibdev->roce_cq_send);
+	ib_destroy_cq(smc_ibdev->roce_cq_recv);
+	flush_work(&smc_ibdev->port_err_work);
+	kfree(smc_ibdev);
+	atomic_dec(&smc_reconfiguring);
+}
+
+struct ib_client smc_ib_client = {
+	.name	= "smc_ib",
+	.add	= smc_ib_add_dev,
+	.remove = smc_ib_remove_dev
+};
+
+void smc_sock_wake_rx(struct sock *sk)
+{
+	struct socket_wq *wq;
+
+	/* derived from sock_def_readable() */
+	/* Attention: sk_data_ready=smc_sock_wake_rx called in listen_worker */
+	rcu_read_lock();
+	wq = rcu_dereference(sk->sk_wq);
+	if (wq_has_sleeper(wq)) {
+		wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLPRI |
+						POLLRDNORM | POLLRDBAND);
+	}
+	if ((sk->sk_shutdown == SHUTDOWN_MASK) ||
+	    (sk->sk_state == SMC_CLOSED))
+		sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_HUP);
+	else
+		sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
+	rcu_read_unlock();
+}
+
+void smc_sock_wake_tx(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct socket *sock = sk->sk_socket;
+	struct socket_wq *wq;
+
+/* similar to sk_stream_write_space */
+	if (atomic_read(&smc->conn.tx_buf_space) && sock) {
+		clear_bit(SOCK_NOSPACE, &sock->flags);
+		rcu_read_lock();
+		wq = rcu_dereference(sk->sk_wq);
+		if (wq_has_sleeper(wq)) {
+			wake_up_interruptible_poll(&wq->wait,
+						   POLLOUT | POLLWRNORM |
+						   POLLWRBAND);
+		}
+		if (wq && wq->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN))
+			sock_wake_async(sock, SOCK_WAKE_SPACE, POLL_OUT);
+		rcu_read_unlock();
+	}
+}
+
+static void smc_sk_release(struct smc_sock *, int);
+static void smc_sock_destruct(struct sock *sk);
+
+static void smc_destruct_non_accepted(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int i;
+
+	sock_hold(sk);
+	lock_sock(sk);
+	smc_sk_release(smc, 0);
+	if (smc->tcpsocket) {
+		struct socket *tcp;
+
+		tcp = smc->tcpsocket;
+		smc->tcpsocket = NULL;
+		sock_release(tcp);
+	}
+	release_sock(sk);
+	if (!smc->use_tcp) {
+		for (i = 0;
+		     ((i < 10) &&
+		      (smc_pending_sends(conn) ||
+		       !smc_close_received(conn)));
+		     i++) {
+			msleep(SMC_WAIT_PENDING_SENDS_TIMEO);
+		}
+	}
+	if (cmpxchg(&smc->sock_put_done, 0, 1) == 0) {
+		sock_set_flag(sk, SOCK_ZAPPED);
+		sock_set_flag(sk, SOCK_DEAD);
+		schedule_work(&smc->release_work);
+	}
+	sock_put(sk);
+}
+
+static void smc_sock_cleanup_listen(struct sock *parent)
+{
+	struct sock *sk;
+
+	/* Close non-accepted connections */
+	while ((sk = smc_accept_dequeue(parent, NULL)))
+		smc_destruct_non_accepted(sk);
+}
+
+static void smc_wait_close(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	signed long timeout;
+	DEFINE_WAIT(wait);
+
+	/* cancel CORK */
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+		schedule_delayed_work(&smc->write_work, 0);
+	}
+	timeout = sk->sk_lingertime ? sk->sk_lingertime : MAX_SCHEDULE_TIMEOUT;
+	if (smc_prepared_sends(&smc->conn) && !sk->sk_err &&
+	    !(current->flags & PF_EXITING)) {
+		release_sock(sk);
+		wait_event_interruptible_timeout(smc->prep_waiter,
+						!smc_prepared_sends(&smc->conn),
+						timeout);
+		lock_sock(sk);
+	}
+	timeout = SMC_WAIT_PENDING_SENDS_TIMEO;
+	if (smc_pending_sends(&smc->conn) &&
+	    !(current->flags & PF_EXITING)) {
+		set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+		do {
+			prepare_to_wait(sk_sleep(sk), &wait,
+					TASK_INTERRUPTIBLE);
+			if (sk_wait_event(sk, &timeout,
+					  !smc_pending_sends(&smc->conn)))
+				break;
+		} while (!signal_pending(current) && timeout);
+		clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+		finish_wait(sk_sleep(sk), &wait);
+	}
+}
+
+static int smc_wait_for_pending_sends(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	u32 i, j;
+	int rc;
+
+	release_sock(&smc->sk);
+	rc = wait_event_timeout(smc->destruct_waiter,
+				!smc_pending_sends(&smc->conn),
+				SMC_WAIT_PENDING_SENDS_TIMEO);
+	lock_sock(&smc->sk);
+	if (!rc) { /* timeout reached */
+		for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+			link = &smc->conn.lgr->lnk[j];
+			if (!link->link_id)
+				continue;
+			for (i = 0; i < link->send_wr_num; i++)
+				if (link->pending[i].conn == &smc->conn) {
+					link->pending[i].conn = NULL;
+					clear_bit(i, link->pending_bit_mask);
+				}
+		}
+	}
+	return rc;
+}
+
+static void smc_conn_term_abnormal(struct smc_sock *smc, int old_state)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+
+	sk_stop_timer(sk, &sk->sk_timer);
+	while (work_busy(&smc->timer_work))
+		msleep(20);
+	if (sk->sk_state == SMC_PEERCLW2 ||
+	    sk->sk_state == SMC_PEERFINCLW ||
+	    sk->sk_state == SMC_CLOSED ||
+	    sk->sk_state == SMC_INIT ||
+	    sk->sk_state == SMC_PEERABORTW) {
+		sk->sk_state = SMC_CLOSED;
+		if (timer_pending(&smc->fin_timer))
+			mod_timer_pending(&smc->fin_timer, jiffies);
+	} else {
+		if (sk->sk_state == SMC_PEERCLW1) {
+			/* enable wait_close wakeup */
+			smc_clear_pending_sends(conn);
+		}
+		/* cancel close waiting */
+		atomic64_set(&conn->tx_curs_prep.s.acurs,
+			     conn->tx_curs_sent.s.lcurs);
+		atomic64_set(&conn->tx_curs_fin.s.acurs,
+			     conn->tx_curs_sent.s.lcurs);
+		sk->sk_state = SMC_PROCESSABORT;
+		/* purge all in-flight data, do abnormal term step */
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+		if ((old_state != SMC_PROCESSABORT) && (conn->lgr) &&
+		    (conn->lgr->lgr_type != NONE)) {
+			smc_ctrl_send(conn, conn->lnk_idx);
+			if (conn->lgr->lgr_type != SINGLE)
+				smc_send_close_alt(smc);
+		}
+		if (sock_flag(sk, SOCK_DEAD))
+			mod_timer_pending(&smc->fin_timer,
+					  jiffies + TCP_TIMEWAIT_LEN);
+	}
+	sk->sk_err = ECONNABORTED;
+}
+
+static void smc_accept_unlink(struct sock *);
+
+void smc_conn_term(struct smc_sock *smc)
+{
+	struct sock *tcpsk;
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int old_state;
+
+	old_state = sk->sk_state;
+	/* Handle abnormal termination */
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		smc_conn_term_abnormal(smc, old_state);
+		goto out;
+	}
+	switch (sk->sk_state) {
+	/* Normal termination - Passive close part */
+	case SMC_INIT:
+	case SMC_ACTIVE:
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done ||
+		    conn->local_rx_ctrl.conn_state_flags.closed_conn) {
+			/* complete any outstanding recv with zero-length
+			 * if peerclosedconn and pending data to be written
+			 * then reset conn
+			 */
+			sk->sk_state = SMC_APPLCLW1;
+			if (sock_flag(sk, SOCK_DEAD))
+				mod_timer(&smc->fin_timer,
+					  jiffies + TCP_TIMEWAIT_LEN);
+		}
+		break;
+	case SMC_PEERFINCLW:
+		if (conn->local_rx_ctrl.conn_state_flags.closed_conn)
+			sk->sk_state = SMC_CLOSED;
+		break;
+		/* Normal termination - Active close part */
+	case SMC_PEERCLW1:
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done) {
+			/* complete any outstanding recv with zero-length */
+			sk->sk_state = SMC_PEERCLW2;
+		} /* fall through */
+	case SMC_PEERCLW2:
+		if (conn->local_rx_ctrl.conn_state_flags.closed_conn) {
+			struct smc_e_ctrl *tx_ctrl = &conn->local_tx_ctrl;
+
+			/* complete any outstanding recv with zero-length */
+			if (sk->sk_shutdown == SHUTDOWN_MASK &&
+			    (tx_ctrl->conn_state_flags.closed_conn ||
+			     tx_ctrl->conn_state_flags.abnormal_close)) {
+				sk->sk_state = SMC_CLOSED;
+				if (timer_pending(&smc->fin_timer))
+					mod_timer_pending(&smc->fin_timer,
+							  jiffies);
+			} else {
+				sk->sk_state = SMC_APPLFINCLW;
+			}
+		}
+		break;
+	default:
+		break;
+	}
+out:
+	if ((sk->sk_err == ECONNABORTED) &&
+	    !list_empty(&smc->accept_q))
+		smc_accept_unlink(sk);
+	if (smc_stop_received(conn)) {
+		sk->sk_shutdown = sk->sk_shutdown | RCV_SHUTDOWN;
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			tcpsk = smc->tcpsocket->sk;
+			tcpsk->sk_shutdown = tcpsk->sk_shutdown | RCV_SHUTDOWN;
+		}
+	}
+	if (smc_close_received(conn) &&
+	    (sk->sk_state == SMC_CLOSED) &&
+	    !sk->sk_socket &&
+	    !smc_pending_sends(conn))
+		sock_set_flag(sk, SOCK_DEAD);
+	if ((old_state != sk->sk_state) &&
+	    (old_state != SMC_INIT))
+		sk->sk_state_change(sk);
+}
+
+static void smc_sk_release(struct smc_sock *smc, int wait)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+
+	if (smc->use_tcp) {
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		goto out;
+	}
+	sk_stop_timer(sk, &sk->sk_timer);
+	cancel_work_sync(&smc->timer_work);
+
+	sk->sk_shutdown = SHUTDOWN_MASK;
+
+	switch (sk->sk_state) {
+	case SMC_INIT:
+	case SMC_CLOSED:
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		break;
+	case SMC_LISTEN:
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			rc = kernel_sock_shutdown(smc->tcpsocket, SHUT_RDWR);
+			smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+			/* to wake up kernel_accept of smc_tcp_listen_worker */
+		}
+		release_sock(sk);
+		smc_sock_cleanup_listen(sk);
+		while (work_busy(&smc->tcp_listen_work) ||
+		       work_busy(&smc->listen_work)) {
+			msleep(20);
+		}
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+		break;
+	case SMC_ACTIVE:
+		/* active close */
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done)
+			sk->sk_state = SMC_PEERCLW2;
+		else
+			sk->sk_state = SMC_PEERCLW1;
+		sk->sk_state_change(sk);
+		/* handle SOCK_LINGER */
+		/* fall through */
+	case SMC_PEERCLW1:
+		if (wait)
+			smc_wait_close(smc);
+		sk->sk_state = SMC_PEERCLW2;
+		smc_send_close(smc);
+		/* fall through */
+	case SMC_PEERCLW2:
+		rc = smc_wait_for_pending_sends(smc);
+		if ((rc <= 0) ||   /* timeout or interrupted */
+		    smc_close_received(conn)) {
+			sk->sk_state = SMC_CLOSED;
+			sk->sk_state_change(sk);
+		}
+		break;
+	case SMC_PROCESSABORT:
+	case SMC_APPLFINCLW:
+		/* socket already shutdown wr or both (active close) */
+		if (!conn->local_tx_ctrl.conn_state_flags.closed_conn &&
+		    conn->lgr) {
+			rc = smc_send_close(smc);
+			smc_wait_for_pending_sends(smc);
+		}
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		break;
+	case SMC_APPLCLW1:
+		/* passive close */
+		if (wait)
+			smc_wait_close(smc);
+		/* fall through */
+	case SMC_APPLCLW2:
+		/* confirm close from peer */
+		smc_send_close(smc);
+		if (smc_close_received(conn)) {
+			smc_wait_for_pending_sends(smc);
+			sk->sk_state = SMC_CLOSED;
+			sk->sk_state_change(sk);
+		} else {
+			sk->sk_state = SMC_PEERFINCLW;
+			sk->sk_state_change(sk);
+		}
+		break;
+	case SMC_PEERFINCLW:
+		break;
+	default:
+		break;
+	}
+out:
+	return;
+}
+
+/* determine vitality of peer
+ * returns 0 if peer is alive
+ */
+int smc_check_peer_gone(struct smc_link_group *lgr)
+{
+	struct smc_link *link;
+	struct llc_qentry *qentry = NULL;
+	int i, rc = 0;
+
+	if (!lgr)
+		return -EPIPE;
+	if (signal_pending(current))
+		return 0;
+	/* check peer with testlink */
+	if (!llc_initiate(lgr, LLC_GRP_TEST_LINK))
+		return -ETIMEDOUT;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		link = &lgr->lnk[i];
+		if (atomic_read(&link->state) != SMC_LINK_UP)
+			continue;
+		rc = llc_send_test_link(link, "KEEPALIVE_CHECK");
+		if (rc)
+			continue;
+		qentry = llc_wait(lgr, LLC_TESTLINK, LLC_WAIT_TIMER_TIMEO,
+				  LLC_TEST_LINK);
+		if (qentry) { /* peer is alive */
+			kfree(qentry);
+			goto out;
+		}
+	}
+	rc = -ETIMEDOUT;
+out:
+	llc_stop(lgr, LLC_GRP_TEST_LINK);
+	return rc;
+}
+
+static int smc_sock_release(struct socket *sock)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc;
+	int rc = 0;
+
+	if (!sk || (sk->sk_state == SMC_DESTRUCT))
+		return rc;
+	smc = smc_sk(sk);
+
+	sock_hold(sk);
+	if (sk->sk_state == SMC_LISTEN)
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+	else
+		lock_sock(sk);
+	smc_sk_release(smc, 1);
+	if (smc->tcpsocket) {
+		struct socket *tcp;
+
+		tcp = smc->tcpsocket;
+		smc->tcpsocket = NULL;
+		sock_release(tcp);
+	}
+
+	/* detach socket */
+	sock->state = SS_UNCONNECTED;
+	sock_set_flag(sk, SOCK_ZAPPED);
+	sock_orphan(sk); /* incl. SOCK_DEAD */
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+	}
+	release_sock(sk);
+	if (!smc->use_tcp && (smc->write_work.work.func == smc_write_worker)) {
+		wake_up_interruptible(&smc->rx_waiter);
+		cancel_delayed_work_sync(&smc->write_work);
+	}
+	if (sk->sk_state == SMC_PEERCLW2) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			/* guarantee socket cleanup */
+			mod_timer_pending(&smc->fin_timer,
+					  jiffies + TCP_FIN_TIMEOUT);
+			smc_terminate_conn(smc->conn.lgr);
+		}
+	}
+
+	/* check if we can cleanup sock or keep it for peers responses */
+	if ((sk->sk_state == SMC_CLOSED) &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		if (timer_pending(&smc->fin_timer))
+			mod_timer_pending(&smc->fin_timer, jiffies);
+		schedule_work(&smc->release_work);
+	}
+	sock->sk = NULL;
+	sock_put(sk);
+
+	return rc;
+}
+
+static int smc_sock_bind(struct socket *sock, struct sockaddr *addr,
+			 int addr_len)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc;
+
+	smc->tcpsocket->sk->sk_reuse = sk->sk_reuse;
+	if (addr->sa_family == AF_SMC)
+		addr->sa_family = AF_INET;
+	rc = kernel_bind(smc->tcpsocket, addr, addr_len);
+	return rc;
+}
+
+/* Wait for data on the tcp-socket and fallback to tcp if decline is received
+ * Returns 0 if success
+ *         tcpsocket error, -ETIMEDOUT, -EPROTO or -ECONNABORTED otherwise
+ */
+int clc_wait_msg(struct smc_sock *smc, int noblock, char *buf, int buflen,
+		 u8 type, int *reason_code)
+{
+	struct sock *sk = smc->tcpsocket->sk;
+	long timeo;
+	int rc = 0;
+	int len;
+	int krflags = 0;
+	struct msghdr msg;
+	struct kvec vec;
+	struct smc_clc_msg_hdr *clcm = (struct smc_clc_msg_hdr *)buf;
+	DEFINE_WAIT(wait);
+
+	timeo = (noblock) ? 0 : LLC_WAIT_FIRST_TIMEO + 1 * HZ;
+	if (noblock)
+		krflags = MSG_DONTWAIT;
+
+again:
+	lock_sock(sk);
+	rc = sk_wait_data(sk, &timeo);
+	release_sock(sk);
+	if (sk->sk_err)
+		rc = -sk->sk_err;
+	if (signal_pending(current)) {
+		rc = -EINTR;
+		sk->sk_err = -rc;
+	}
+	if (rc < 0) {
+		smc->sk.sk_err = sk->sk_err;
+		goto out;
+	}
+	if ((rc == 0) && !noblock) {
+		if (timeo) {
+			goto again;
+		} else {
+			smc->sk.sk_err = ETIMEDOUT;
+			rc = -ETIMEDOUT;
+			goto out;
+		}
+	}
+	rc = 0;
+	vec.iov_base = buf;
+	vec.iov_len = buflen;
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_recvmsg(smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_clc_msg_hdr), krflags);
+	if (!len) { /* no data */
+		smc->sk.sk_err = ETIMEDOUT;
+		rc = -ETIMEDOUT;
+		goto out;
+	}
+	if (len < sizeof(struct smc_clc_msg_hdr)) {
+		smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	len = ntohs(clcm->length);
+	if (!len) {
+		smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	vec.iov_base = buf + sizeof(struct smc_clc_msg_hdr);
+	vec.iov_len = buflen - sizeof(struct smc_clc_msg_hdr);
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_recvmsg(smc->tcpsocket, &msg, &vec, 1,
+			     len - sizeof(struct smc_clc_msg_hdr), 0);
+	if (len <= 0) {
+		smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	len = len + sizeof(struct smc_clc_msg_hdr);
+	if ((len < sizeof(struct smc_decline_clc_msg)) ||
+	    memcmp(clcm->ec_clce, SMC_EC, sizeof(SMC_EC))) {
+		if (!sk->sk_err)
+			sk->sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	if (clcm->type == SMC_CLC_DECLINE) {
+		smc->use_tcp = 1;
+		if (clcm->flags & SMC_LINK_GROUP_OUT_OF_SYNCH) {
+			smc_terminate_conn(smc->conn.lgr);
+			sk->sk_err = ECONNABORTED;
+			rc = -ECONNABORTED;
+			return rc;
+		}
+		*reason_code = SMC_CLC_DEC_REPLY;
+		goto out;
+	}
+	if ((clcm->type != type) ||
+	    (memcmp(&clcm->ec_clce, SMC_EC, sizeof(SMC_EC)))) {
+		if (!smc->sk.sk_err)
+			smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+out:
+	return rc;
+}
+
+int clc_send_decline(struct smc_sock *smc, u32 reason_code, u8 flags)
+{
+	struct smc_decline_clc_msg dclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int len;
+
+	memset(&dclc, 0, sizeof(struct smc_decline_clc_msg));
+	memcpy(dclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(dclc.peer_id, local_peerid, sizeof(local_peerid));
+	dclc.hdr.type = SMC_CLC_DECLINE;
+	dclc.hdr.length = htons(sizeof(struct smc_decline_clc_msg));
+	dclc.hdr.flags = SMC_CLC_V1 | flags;
+	dclc.peer_diagnosis = htonl(reason_code);
+	memcpy(dclc.ec_dclcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &dclc;
+	vec.iov_len = sizeof(struct smc_decline_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_decline_clc_msg));
+	if (len < sizeof(struct smc_decline_clc_msg))
+		smc->sk.sk_err = EPROTO;
+	if (len < 0)
+		smc->sk.sk_err = -len;
+	return len;
+}
+
+/* Copy flags relevant for SO_* sockopts */
+static void smc_copy_sockopt_settings(struct sock *nsk, struct sock *osk)
+{
+	nsk->sk_type = osk->sk_type;
+	nsk->sk_sndbuf = osk->sk_sndbuf;
+	nsk->sk_rcvbuf = osk->sk_rcvbuf;
+	nsk->sk_sndtimeo = osk->sk_sndtimeo;
+	nsk->sk_rcvtimeo = osk->sk_rcvtimeo;
+	nsk->sk_mark = osk->sk_mark;
+	nsk->sk_priority = osk->sk_priority;
+	nsk->sk_rcvlowat = osk->sk_rcvlowat;
+	nsk->sk_bound_dev_if = osk->sk_bound_dev_if;
+	sock_copy_flags(nsk, osk);
+}
+
+static void smc_init_tmp_sockopts(struct smc_sock *smc)
+{
+	smc->tmp_sockopts.set = 0;
+	smc->tmp_sockopts.cork = 0;
+	smc->tmp_sockopts.defer_accept = 0;
+	smc->tmp_sockopts.nodelay = 0;
+}
+
+static int smc_sock_setsockopt(struct socket *, int, int, char __user *,
+			       unsigned int);
+
+/* Apply all user-set sockopts effective that were supressed as they
+ * might have a negative effect during the CLC handshake
+ */
+static int smc_apply_tmp_sockopts(struct socket *socket,
+				  const struct smc_tmp_sockopts *sopts)
+{
+	int rc = 0, brc;
+	mm_segment_t old_fs = get_fs();
+
+	set_fs(KERNEL_DS);
+	if (sopts->set & SMC_SOCKOPTS_CORK) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_CORK,
+					  (char __user *)&sopts->cork,
+					  sizeof(sopts->cork));
+		if (brc)
+			rc = -EFAULT;
+	}
+	if (sopts->set & SMC_SOCKOPTS_DEFER_ACCEPT) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_DEFER_ACCEPT,
+					  (char __user *)&sopts->defer_accept,
+					  sizeof(sopts->defer_accept));
+		if (brc)
+			rc = -EFAULT;
+	}
+	if (sopts->set & SMC_SOCKOPTS_NODELAY) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_NODELAY,
+					  (char __user *)&sopts->nodelay,
+					  sizeof(sopts->nodelay));
+		if (brc)
+			rc = -EFAULT;
+	} else {
+		struct smc_sock *smc = smc_sk(socket->sk);
+
+		tcp_sk(smc->tcpsocket->sk)->nonagle &= ~TCP_NAGLE_OFF;
+	}
+	set_fs(old_fs);
+
+	return rc;
+}
+
+static int smc_find_link_accept(struct smc_link_group *lgr,
+				struct smc_acc_conf_clc_msg *clc)
+{
+	int i;
+	u32 tmp_qpnum;
+
+	ntoh_three(&tmp_qpnum, clc->qp_num);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (!memcmp(&lgr->lnk[i].gid_peer, &clc->hdr2.gid, 16) &&
+		    !memcmp(&lgr->lnk[i].mac_peer, &clc->hdr2.mac, 6) &&
+		    (lgr->lnk[i].qp_num_peer == tmp_qpnum))
+			return i;
+	}
+	return -ENOENT;
+}
+
+static int smc_rtoken_handling(struct smc_link_group *lgr, struct smc_sock *smc,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	struct smc_connection *conn = &smc->conn;
+	u64 vaddr = be64_to_cpu(clc->rmb_vaddr);
+	unsigned int rkey = ntohl(clc->rmb_rkey);
+	int lnk_idx;
+
+	lnk_idx = smc_find_link_accept(lgr, clc);
+	if (lnk_idx < 0)
+		return -ENOENT;
+	conn->rtok_idx =
+		smc_find_rtoken_by_link(lgr, lnk_idx, rkey);
+	if (conn->rtok_idx == -ENOENT) {
+		conn->rtok_idx = smc_get_rtoken(lgr);
+		lgr->rtok[conn->rtok_idx].rkey[lnk_idx] = rkey;
+		lgr->rtok[conn->rtok_idx].vaddr[lnk_idx] = vaddr;
+		lgr->rtok[conn->rtok_idx].link_id[lnk_idx] =
+			lgr->lnk[lnk_idx].link_id;
+	}
+	return 0;
+}
+
+static int smc_send_proposal(struct smc_sock *smc,
+			     struct smc_roce_defs *roce_sugg,
+			     struct smc_acc_conf_clc_msg *aclc,
+			     int *reason_code)
+{
+	int pclc_msgsize = sizeof(struct smc_proposal_clc_msg) +
+			   sizeof(struct smc_proposal_clc_msg2);
+	struct smc_proposal_clc_msg *pclc;
+	struct smc_proposal_clc_msg2 *pclc2;
+	struct kvec vec;
+	struct msghdr msg;
+	int rc = 0;
+
+	pclc = kzalloc(pclc_msgsize, GFP_KERNEL);
+	if (!pclc)
+		return -ENOMEM;
+	/* send SMC Proposal CLC message (peer ID, GID, MAC, VLAN) */
+	memcpy(&pclc->hdr2.gid, &roce_sugg->gid, sizeof(roce_sugg->gid));
+	memcpy(&pclc->hdr2.mac, &roce_sugg->mac, sizeof(roce_sugg->mac));
+	pclc->iparea_offset = 0;
+
+	pclc2 = (struct smc_proposal_clc_msg2 *)
+		((char *)pclc + sizeof(struct smc_proposal_clc_msg) +
+		 pclc->iparea_offset);
+
+	/* is it sufficient to use IP from connection itself? */
+	rc = smc_netinfo_by_tcpsk(&pclc2->outgoing_subnet,
+				  &pclc2->subnet_mask_bits, smc->tcpsocket);
+	memcpy(&pclc->hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(&pclc->hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	pclc->hdr.type = SMC_CLC_PROPOSAL;
+	pclc->hdr.length = htons(pclc_msgsize);
+	pclc->hdr.flags = SMC_CLC_V1;
+	memcpy(&pclc2->ec_pclcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = pclc;
+	vec.iov_len = pclc_msgsize;
+	memset(&msg, 0, sizeof(struct msghdr));
+	rc = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1, pclc_msgsize);
+	kfree(pclc);
+	if (rc >= 0) {
+		if (rc < pclc_msgsize)
+			rc = -EPROTO;
+		else
+			rc = 0;
+	}
+	if (rc) {
+		smc->sk.sk_err = -rc;
+		return rc;
+	}
+
+	/* receive SMC Accept CLC message (GID, MAC, VLAN, QP )*/
+	rc = clc_wait_msg(smc, SMC_CLC_BLOCK, (char *)aclc,
+			  sizeof(struct smc_acc_conf_clc_msg),
+			  SMC_CLC_ACCEPTANCE, reason_code);
+	return rc;
+}
+
+static int smc_send_confirm(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct smc_connection *conn = &smc->conn;
+	struct smc_acc_conf_clc_msg cclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int rc = 0;
+
+	link = &conn->lgr->lnk[conn->lnk_idx];
+	lgr = link->lnk_grp;
+	/* send SMC Confirm CLC msg (MAC, GID, QP, link num, link user id) */
+	memset(&cclc, 0, sizeof(struct smc_acc_conf_clc_msg));
+	cclc.tcp_conn_idx = 1; /* as long as 1 RMB = 1 RMBE */
+	memcpy(cclc.hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	memcpy(&cclc.hdr2.gid, &link->roce_defs.gid,
+	       sizeof(link->roce_defs.gid));
+	memcpy(&cclc.hdr2.mac, &link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+	hton_three(link->roce_qp->qp_num, cclc.qp_num);
+	hton_three(link->psn_initial, cclc.psn);
+	cclc.rmb_rkey =
+		htonl(conn->rmb_rx_elem->mr_rx[conn->lnk_idx]->rkey);
+	cclc.rmb_vaddr =
+		cpu_to_be64((u64)conn->rmb_rx_elem->rmb_rx_dma[conn->lnk_idx]);
+	memcpy(cclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	cclc.hdr.type = SMC_CLC_CONFIRMATION;
+	cclc.hdr.length = htons(sizeof(struct smc_acc_conf_clc_msg));
+	cclc.hdr.flags = SMC_CLC_V1;
+	cclc.rmb_alert_token = conn->alert_token_local;
+	cclc.flags2 = conn->rmb_rx_size_short +
+		      min(link->path_mtu, link->mtu_peer);
+	memcpy(cclc.ec_clcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &cclc;
+	vec.iov_len = sizeof(struct smc_acc_conf_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	rc = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1,
+			    sizeof(struct smc_acc_conf_clc_msg));
+	if (rc >= 0) {
+		if (rc < sizeof(struct smc_acc_conf_clc_msg))
+			rc = -EPROTO;
+		else
+			rc = 0;
+	}
+	if (rc)
+		smc->sk.sk_err = -rc;
+	return rc;
+}
+
+static int smc_initial_map_rmb(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &lgr->lnk[smc->conn.lnk_idx];
+	long rc = 0;
+
+	if (!smc->conn.tx_rmb_reused) {
+		rc = smc_map_rmbs_tx(lgr, smc->conn.lnk_idx,
+				     smc->conn.rmb_tx_size,
+				     smc->conn.rmb_tx_elem);
+		if (rc)
+			goto out;
+	}
+	if (!smc->conn.rx_rmb_reused) {
+		rc = smc_map_rmbs_rx(lgr, smc->conn.lnk_idx,
+				     smc->conn.rmb_rx_size,
+				     smc->conn.rmb_rx_elem);
+		if (rc)
+			goto out;
+		rc = smc_get_dma_mr(lgr, smc->conn.lnk_idx,
+				    smc->conn.rmb_rx_elem);
+		if (rc)
+			goto out;
+	}
+	return 0;
+
+out:
+	if (rc && smc_lnk_downing(&link->state)) {
+		release_sock(&smc->sk);
+		smc_link_down(lgr, link - &lgr->lnk[0]);
+		lock_sock(&smc->sk);
+	}
+	return (int)rc;
+}
+
+static int smc_announce_rmb(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* Announce new RMB */
+	if (smc->conn.tx_rmb_reused && smc->conn.rx_rmb_reused)
+		goto out;
+	if (!llc_initiate(lgr, LLC_GRP_CONF_RKEY)) { /* timeout */
+		*reason_code = SMC_CLC_DEC_TU;
+		rc = -ETIMEDOUT;
+		goto out;
+	}
+	smc_map_rmbs(smc);
+	if (lgr->lgr_type == SINGLE)
+		goto out_stop;
+	if (smc->conn.rx_rmb_reused)
+		goto out_stop;
+	rc = llc_do_confirm_rkey(link, smc->conn.rmb_rx_elem);
+	if (rc >= 0) {
+		if (lgr->role == SMC_CLNT)
+			release_sock(&smc->sk);
+		qentry = llc_wait(lgr, lgr->role, LLC_WAIT_TIMEO,
+				  LLC_CONFIRM_RKEY);
+		if (lgr->role == SMC_CLNT)
+			lock_sock(&smc->sk);
+		if ((!qentry) ||
+		    (qentry->msg.hd.flags & LLC_FLAG_RKEY_NEG)) {
+			kfree(qentry);
+			if (smc_lnk_downing(&link->state)) {
+				if (lgr->role == SMC_CLNT)
+					release_sock(&smc->sk);
+				smc_link_down(lgr, link - &lgr->lnk[0]);
+				if (lgr->role == SMC_CLNT)
+					lock_sock(&smc->sk);
+			}
+			*reason_code = SMC_CLC_DEC_TU;
+			rc = -ETIMEDOUT;
+			goto out_stop;
+		}
+		kfree(qentry);
+		rc = 0;
+	} else {
+		*reason_code = SMC_CLC_DEC_SEND;
+		rc = -EPIPE;
+	}
+out_stop:
+	llc_stop(lgr, LLC_GRP_CONF_RKEY);
+out:
+	if (rc)
+		smc_remove_rx_rmb(smc);
+	return rc;
+}
+
+static int smc_clnt_conf_first_lnk(struct smc_sock *smc,
+				   struct smc_roce_defs *roce_sugg,
+				   int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_confirm_msg *conf_llc;
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* receive CONFIRM LINK request over RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_CONFIRM_LINK);
+	if (!qentry) {
+		/* clc decl received or llc timeout */
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_NO_NOTIFY, smc);
+		return rc;
+	}
+	if (!(qentry->msg.hd.flags & 0xf)) {
+		/* no cache alignment specified, decline */
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		*reason_code = SMC_CLC_DEC_INT;
+		rc = -EOPNOTSUPP;
+		kfree(qentry);
+		return rc;
+	}
+	lgr->cache_align_size =
+		0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+	conf_llc = (struct llc_confirm_msg *)&qentry->msg;
+	link->link_id = conf_llc->link_num;
+	lgr->rtok[smc->conn.rtok_idx].link_id[smc->conn.lnk_idx] =
+		link->link_id;
+	lgr->max_links = SMC_MAX_SYM_LINKS;
+	kfree(qentry);
+	rc = smc_modify_qp_rts(link);
+	if (rc) {
+		*reason_code = SMC_CLC_DEC_INT;
+		return rc;
+	}
+	smc_check_qp_attr(link);
+	/* send CONFIRM LINK response over RoCE fabric */
+	rc = llc_send_confirm_link(link, (u8 *)link->roce_defs.mac,
+				   &roce_sugg->gid, LLC_RESP, LLC_SERV);
+	if (rc < 0) {
+		*reason_code = SMC_CLC_DEC_TCL;
+		return rc;
+	}
+	atomic_set(&link->state, SMC_LINK_UP);
+	return rc;
+}
+
+static int smc_clnt_add_lnk(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	release_sock(&smc->sk);
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_ADD_LINK);
+	lock_sock(&smc->sk);
+	if (!qentry) {
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_SERV, smc);
+		return rc;
+	}
+	rc = llc_cli_add_link(link, qentry, smc);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+	if (rc == -ENOLINK)
+		*reason_code = SMC_LLC_DEL_NOLNK;
+	return rc;
+}
+
+static void smc_save_peer_info(struct smc_sock *smc,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	smc->conn.local_tx_ctrl.ctrl_token = clc->rmb_alert_token;
+	smc->conn.peer_rx_buf_len =
+		smc_uncompress_bufsize(clc->flags2 & 0xf0);
+	smc->conn.peer_tcp_conn_idx = clc->tcp_conn_idx;
+	atomic_set(&smc->conn.rx_buf_space, smc->conn.peer_rx_buf_len);
+}
+
+static void smc_save_link_info(struct smc_link *link,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	memcpy(link->gid_peer, &clc->hdr2.gid, sizeof(link->gid_peer));
+	memcpy(link->mac_peer, clc->hdr2.mac, sizeof(link->mac_peer));
+	ntoh_three(&link->psn_peer, clc->psn);
+	link->mtu_peer = clc->flags2 & 0x0f;
+}
+
+static int smc_switch_to_tcp(struct smc_sock *smc, int reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	int rc;
+
+	smc->use_tcp = 1;
+	if (reason_code && (reason_code != SMC_CLC_DEC_REPLY)) {
+		rc = clc_send_decline(smc, reason_code, 0);
+		if (rc < sizeof(struct smc_decline_clc_msg))
+			return rc;
+	}
+	if (lgr) {
+		if (smc->new_lgr && link) {
+			atomic_set(&link->state, SMC_LINK_DOWN);
+			lgr->lgr_type = NONE;
+		}
+		if (lgr->lgr_type == NONE) {
+			spin_lock_bh(&smc_lgr_list.lock);
+			list_del_init(&lgr->list);
+			spin_unlock_bh(&smc_lgr_list.lock);
+		}
+	}
+	smc_free_conn(smc);
+	return 0;
+}
+
+static void smc_init_after_clc_handshake(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	u32 keepalive_time;
+
+	init_waitqueue_head(&smc->prep_waiter);
+	init_waitqueue_head(&smc->destruct_waiter);
+	if (sk->sk_state == SMC_INIT)
+		sk->sk_state = SMC_ACTIVE;
+	if (smc->use_tcp)
+		goto out;
+	INIT_DELAYED_WORK(&smc->write_work, smc_write_worker);
+	if (sock_flag(sk, SOCK_KEEPOPEN)) {
+		keepalive_time = tp->keepalive_time ? : TCP_KEEPALIVE_TIME;
+		if (keepalive_time)
+			sk_reset_timer(sk, &sk->sk_timer,
+				       jiffies + (unsigned long)keepalive_time);
+	}
+	sk->sk_data_ready = smc_sock_wake_rx;
+	sk->sk_write_space = smc_sock_wake_tx;
+	/* to return meaningful values with getsockopt */
+	sk->sk_sndbuf = smc->conn.rmb_tx_size;
+	sk->sk_rcvbuf = smc->conn.rmb_rx_size;
+out:
+	return;
+}
+
+static int smc_connect(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct socket *sock = sk->sk_socket;
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	int rc = 0;
+	struct sockaddr_in *inaddr =  (struct sockaddr_in *)smc->addr;
+	struct smc_acc_conf_clc_msg aclc;
+	struct smc_link_group *lgr = NULL;
+	struct smc_link *link = NULL;
+	struct smc_roce_defs roce_sugg;
+	u32 tmp_qpnum;
+	char pnet_id[SMC_MAX_PNET_ID_LEN];
+	int reason_code = 0;
+
+	/* check if peer is smc capable */
+	if (!tp->syn_smc) {
+		smc->use_tcp = 1;
+		goto out_connected;
+	}
+
+	smc_find_roce_resources(&roce_sugg, pnet_id, inaddr,
+				smc->tcpsocket->sk);
+	if (!roce_sugg.ibdev) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+	/* do inband token exchange */
+	rc = smc_send_proposal(smc, &roce_sugg, &aclc, &reason_code);
+	if (rc) {
+		if ((rc == -EPROTO) || (rc == -ETIMEDOUT)) {
+			smc->sk.sk_err = ENETUNREACH;
+			rc = -ENETUNREACH;
+		}
+		goto out_err;
+	}
+	if (reason_code) { /* switch to tcp */
+		smc->use_tcp = 1;
+		goto out_connected;
+	}
+
+	/* allocate connection/link group*/
+	ntoh_three(&tmp_qpnum, aclc.qp_num);
+	smc->new_lgr = aclc.hdr.flags & SMC_FIRST_CONTACT;
+	rc = smc_create_conn(smc, inaddr->sin_addr.s_addr, SMC_CLNT,
+			     &roce_sugg, &aclc.hdr2, tmp_qpnum);
+	if (rc == -ENOLINK) { /* link groups out of sync */
+		smc->sk.sk_err = ENETUNREACH;
+		rc = -ENETUNREACH;
+		goto out_err;
+	}
+	if (rc) {
+		/* insufficient memory */
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	lgr = smc->conn.lgr;
+	link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	strncpy(lgr->pnet_id, pnet_id, sizeof(pnet_id));
+	smc_save_peer_info(smc, &aclc);
+	lock_sock(sk);
+
+	rc = smc_create_rmbs(smc);
+	if (rc) {
+		/* insufficient memory */
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	atomic_set(&smc->conn.tx_buf_space, smc->conn.rmb_tx_size);
+	if (smc->new_lgr) {
+		rc = smc_initial_map_rmb(smc);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_RIM;
+			goto decline;
+		}
+		link->qp_num_peer = tmp_qpnum;
+		smc_save_link_info(link, &aclc);
+	}
+	rc = smc_rtoken_handling(lgr, smc, &aclc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_INT;
+		goto decline;
+	}
+	if (smc->new_lgr) {
+		rc = smc_ready_link(link);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_INT;
+			goto decline;
+		}
+		llc_initiate(lgr, LLC_GRP_ADD_LINK);
+		lgr->lgr_type = SINGLE;
+	} else { /* link already active */
+		rc = smc_announce_rmb(smc, &reason_code);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_TU;
+			goto decline;
+		}
+	}
+	rc = smc_send_confirm(smc);
+	if (rc)
+		goto out_err;
+
+	if (smc->new_lgr) {
+		/* QP confirmation over RoCE fabric */
+		rc = smc_clnt_conf_first_lnk(smc, &roce_sugg, &reason_code);
+		if (rc == -EINTR)
+			goto out_err;
+		if (rc)
+			goto decline;
+		/* receive ADD LINK request over RoCE fabric */
+		rc = smc_clnt_add_lnk(smc, &reason_code);
+		if (rc == -ENOLINK)
+			goto out_connected;
+		if (rc == -ETIMEDOUT)
+			goto out_err;
+		if (rc)
+			goto decline;
+	}
+
+out_connected:
+	sock->state = SS_CONNECTED;
+	smc_init_after_clc_handshake(smc);
+	/* copy setsockopt settings to tcpsocket to be prepared for fallback */
+	smc_copy_sockopt_settings(smc->tcpsocket->sk, sk);
+	/* to return meaningful values with getsockopt */
+	if (lgr) {
+		release_sock(sk);
+		if (mutex_is_locked(&lgr->conf_mutex))
+			mutex_unlock(&lgr->conf_mutex);
+	}
+	smc_apply_tmp_sockopts(sk->sk_socket, &smc->tmp_sockopts);
+	return 0;
+
+decline:
+	rc = smc_switch_to_tcp(smc, reason_code);
+	if (!rc)
+		goto out_connected;
+out_err:
+	if (lgr) {
+		if (smc->new_lgr)
+			lgr->lgr_type = NONE;
+		release_sock(sk);
+		if (mutex_is_locked(&lgr->conf_mutex))
+			mutex_unlock(&lgr->conf_mutex);
+		smc_free_conn(smc);
+	}
+	smc->use_tcp = 1;
+	sk->sk_state = SMC_CLOSED;
+	return rc;
+}
+
+static int smc_sock_connect(struct socket *sock, struct sockaddr *addr,
+			    int alen, int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	int rc, loop = 0;
+
+	/* copy setsockopt settings to tcpsocket */
+	smc_copy_sockopt_settings(smc->tcpsocket->sk, sk);
+	/* set SMC capability on tcp-socket */
+	tp->syn_smc = 1;
+	smc->addr = addr;
+	if (addr->sa_family == AF_SMC)
+		addr->sa_family = AF_INET;
+
+again:
+	rc = kernel_connect(smc->tcpsocket, addr, alen, flags);
+	if (rc == -ERESTARTSYS) {
+		msleep(1000);
+		loop++;
+		if (loop < 10)
+			goto again;
+		rc = -EAGAIN;
+	}
+	if (rc == -EINPROGRESS)
+		return rc;
+
+	if (!rc) {
+		rc = smc_connect(smc);
+		if (rc)
+			rc = -1;
+	} else {
+		smc->use_tcp = 1;
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_err = -rc;
+		sk->sk_state_change(sk);
+		rc = -1;
+	}
+
+	return rc;
+}
+
+static int smc_sock_listen(struct socket *sock, int backlog)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	int rc;
+
+	lock_sock(sk);
+	/* copy setsockopt settings to tcpsocket */
+	smc_copy_sockopt_settings(smc->tcpsocket->sk, sk);
+	/* set SMC capability on tcp-socket */
+	tp->syn_smc = 1;
+	rc = kernel_listen(smc->tcpsocket, backlog);
+
+	if (!rc) {
+		sk->sk_max_ack_backlog = backlog;
+		sk->sk_ack_backlog = 0;
+		sk->sk_state = SMC_LISTEN;
+		if (!smc->use_tcp) {
+			INIT_WORK(&smc->tcp_listen_work, smc_tcp_listen_worker);
+			schedule_work(&smc->tcp_listen_work);
+		}
+	}
+
+	release_sock(sk);
+	return rc;
+}
+
+static void smc_accept_enqueue(struct sock *parent, struct sock *sk)
+{
+	struct smc_sock *par = smc_sk(parent);
+
+	sock_hold(sk);
+	spin_lock(&par->accept_q_lock);
+	list_add_tail(&smc_sk(sk)->accept_q, &par->accept_q);
+	spin_unlock(&par->accept_q_lock);
+	sk_acceptq_added(parent);
+}
+
+static void smc_accept_unlink(struct sock *sk)
+{
+	struct smc_sock *par = smc_sk(sk)->listen_smc;
+
+	spin_lock(&par->accept_q_lock);
+	list_del_init(&smc_sk(sk)->accept_q);
+	spin_unlock(&par->accept_q_lock);
+	sk_acceptq_removed(&smc_sk(sk)->listen_smc->sk);
+	sock_put(sk);
+}
+
+static struct sock *smc_accept_dequeue(struct sock *parent,
+				       struct socket *newsock)
+{
+	struct smc_sock *isk, *n;
+	struct sock *sk;
+
+	list_for_each_entry_safe(isk, n, &smc_sk(parent)->accept_q, accept_q) {
+		sk = (struct sock *)isk;
+		lock_sock(sk);
+
+		smc_accept_unlink(sk);
+		if (newsock)
+			sock_graft(sk, newsock);
+		release_sock(sk);
+		return sk;
+	}
+	return NULL;
+}
+
+static int smc_sock_accept(struct socket *sock, struct socket *newsock,
+			   int flags)
+{
+	DECLARE_WAITQUEUE(wait, current);
+	struct sock *sk = sock->sk, *nsk;
+	long timeo;
+	int rc = 0;
+
+	lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+	timeo = sock_rcvtimeo(sk, flags & O_NONBLOCK);
+
+	/* Wait for an incoming connection */
+	add_wait_queue_exclusive(sk_sleep(sk), &wait);
+	while (!(nsk = smc_accept_dequeue(sk, newsock))) {
+		set_current_state(TASK_INTERRUPTIBLE);
+		if (!timeo) {
+			rc = -EAGAIN;
+			break;
+		}
+
+		release_sock(sk);
+		timeo = schedule_timeout(timeo);
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+
+		if (signal_pending(current)) {
+			rc = -EINTR;
+			break;
+		}
+	}
+
+	set_current_state(TASK_RUNNING);
+	remove_wait_queue(sk_sleep(sk), &wait);
+
+	if (rc) {
+		release_sock(sk);
+		goto done;
+	}
+	rc = sock_error(nsk);
+	if (rc) {
+		release_sock(sk);
+		goto done;
+	}
+
+	newsock->state = SS_CONNECTED;
+	release_sock(sk);
+	smc_apply_tmp_sockopts(newsock, &smc_sk(sk)->tmp_sockopts);
+
+	/* don't get value from tmp_sockopts as TCP will mangle the value */
+	timeo = 0;
+	if (smc_sk(sk)->tmp_sockopts.set & SMC_SOCKOPTS_DEFER_ACCEPT)
+		timeo = smc_getsockopt_from_tcp(smc_sk(nsk), TCP_DEFER_ACCEPT);
+
+	if (!rc && timeo && !(flags & O_NONBLOCK)) {
+		int rc = 0;
+
+		timeo = msecs_to_jiffies(timeo * 1000);
+		if (smc_sk(nsk)->use_tcp) {
+			lock_sock(smc_sk(nsk)->tcpsocket->sk);
+			rc = sk_wait_data(smc_sk(nsk)->tcpsocket->sk, &timeo);
+			release_sock(smc_sk(nsk)->tcpsocket->sk);
+		} else {
+			lock_sock(nsk);
+			rc = smc_wait_rx_data(smc_sk(nsk), 1, 1, timeo);
+			release_sock(nsk);
+		}
+	}
+done:
+	return rc;
+}
+
+static int smc_getsockopt_from_tcp(struct smc_sock *smc, int optname)
+{
+	char optval[10];
+	int optlen;
+	int rc;
+
+	optlen = sizeof(optlen);
+	memset(optval, 0, sizeof(optval));
+	rc = kernel_getsockopt(smc->tcpsocket, SOL_TCP, optname, optval,
+			       &optlen);
+	if (rc)
+		return -EINVAL;
+	memcpy(&rc, optval, optlen);
+	return rc;
+}
+
+static void smc_keepalive_timer(unsigned long);
+static void smc_sock_destruct(struct sock *);
+
+static int smc_send_accept(struct smc_sock *new_smc,
+			   struct smc_acc_conf_clc_msg *cclc, int *reason_code)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct smc_connection *conn = &new_smc->conn;
+	struct smc_acc_conf_clc_msg aclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int len, rc = 0;
+
+	link = &conn->lgr->lnk[conn->lnk_idx];
+	lgr = link->lnk_grp;
+	/* send SMC Accept CLC message (GID, MAC, VLAN, QP) */
+	memset(&aclc, 0, sizeof(struct smc_acc_conf_clc_msg));
+	aclc.hdr.flags = SMC_CLC_V1; /* smc version */
+	if (new_smc->new_lgr)
+		aclc.hdr.flags |= SMC_FIRST_CONTACT;
+	aclc.tcp_conn_idx = 1; /* as long as 1 RMB = 1 RMBE */
+	memcpy(&aclc.hdr2.gid, &link->roce_defs.gid,
+	       sizeof(link->roce_defs.gid));
+	memcpy(&aclc.hdr2.mac, link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+	hton_three(link->roce_qp->qp_num, aclc.qp_num);
+	hton_three(link->psn_initial, aclc.psn);
+	aclc.rmb_rkey =
+		htonl(conn->rmb_rx_elem->mr_rx[conn->lnk_idx]->rkey);
+	aclc.rmb_vaddr =
+		cpu_to_be64((u64)conn->rmb_rx_elem->rmb_rx_dma[conn->lnk_idx]);
+	memcpy(aclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(aclc.hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	aclc.hdr.type = SMC_CLC_ACCEPTANCE;
+	aclc.hdr.length = htons(sizeof(struct smc_acc_conf_clc_msg));
+	aclc.rmb_alert_token = conn->alert_token_local;
+	aclc.flags2 = conn->rmb_rx_size_short + link->path_mtu;
+	memcpy(aclc.ec_clcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &aclc;
+	vec.iov_len = sizeof(struct smc_acc_conf_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_sendmsg(new_smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_acc_conf_clc_msg));
+	if (len < sizeof(struct smc_acc_conf_clc_msg)) {
+		*reason_code = SMC_CLC_DEC_SEND;
+		goto out;
+	}
+
+	/* receive SMC Confirm CLC message (MAC, GID, QP, link #, link user id)
+	 */
+	rc = clc_wait_msg(new_smc, SMC_CLC_BLOCK, (char *)cclc,
+			  sizeof(struct smc_acc_conf_clc_msg),
+			  SMC_CLC_CONFIRMATION, reason_code);
+out:
+	return rc;
+}
+
+static int smc_serv_conf_first_lnk(struct smc_sock *smc,
+				   struct smc_roce_defs *roce_sugg,
+				   int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* send CONFIRM LINK request over the RoCE fabric */
+	llc_initiate(lgr, LLC_GRP_ADD_LINK);
+	rc = llc_send_confirm_link(link, (u8 *)link->roce_defs.mac,
+				   &roce_sugg->gid, LLC_REQ, LLC_SERV);
+	if (rc < 0) {
+		*reason_code = SMC_CLC_DEC_TCL;
+		lgr->lgr_type = NONE;
+		goto out;
+	}
+	/* receive CONFIRM LINK response over the RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_CONFIRM_LINK);
+	if (!qentry) {
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_SERV, smc);
+		atomic_set(&link->state, SMC_LINK_DOWN);
+		lgr->lgr_type = NONE;
+		if (rc == -ECOMM)
+			*reason_code = SMC_CLC_DEC_SEND;
+		if (!rc) /* clc decline received */
+			rc = -ECANCELED;
+	} else {
+		atomic_set(&link->state, SMC_LINK_UP);
+		if (!(qentry->msg.hd.flags & 0xf)) {
+			/* no cache alignment specified, decline */
+			kfree(qentry);
+			*reason_code = SMC_CLC_DEC_REPLY; /* no clc decline */
+			llc_send_del_link(link, link, LLC_FLAG_DEL_ORDERLY,
+					  LLC_DEL_PROT_VIOL, LLC_NO_NOTIFY);
+			atomic_set(&link->state, SMC_LINK_DOWN);
+			lgr->lgr_type = NONE;
+			schedule_delayed_work(&lgr->free_lgr_work,
+					      SMC_FREE_LGR_TIMEO);
+			rc = -EOPNOTSUPP;
+		} else {
+			lgr->cache_align_size = 0x00000001 <<
+				(((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+			kfree(qentry);
+		}
+	}
+out:
+	return rc;
+}
+
+static int smc_serv_add_lnk(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct smc_roce_defs roce_sugg;
+	int rc = 0;
+
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	roce_sugg.ibdev = NULL;
+	roce_sugg.port = 0;
+	roce_sugg.vlan = lgr->vlan;
+	smc_find_alt_roce_resources(&roce_sugg, lgr, link);
+	rc = llc_srv_add_link(link, &roce_sugg, smc);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+	return rc;
+}
+
+static void smc_listen_worker(struct work_struct *work)
+{
+	struct smc_sock *new_smc = container_of(work, struct smc_sock,
+						listen_work);
+	struct sock *newsmcsk = &new_smc->sk;
+	struct socket *newtcpsocket = new_smc->tcpsocket;
+	struct tcp_sock *ntp = tcp_sk(newtcpsocket->sk);
+	struct smc_sock *lsmc = new_smc->listen_smc;
+	struct sockaddr_in peeraddr;
+	char pclc_buf[sizeof(struct smc_proposal_clc_msg)
+		    + sizeof(struct smc_proposal_clc_msg2) + 40];
+	struct smc_proposal_clc_msg *pclc =
+		(struct smc_proposal_clc_msg *)&pclc_buf;
+	struct smc_proposal_clc_msg2 *pclc2;
+	struct smc_acc_conf_clc_msg cclc;
+	struct smc_link_group *lgr = NULL;
+	struct smc_link *link = NULL;
+	struct smc_roce_defs roce_sugg;
+	__be32 subnet = 0;
+	int reason_code = 0;
+	int rc = 0, len;
+	char pnet_id[SMC_MAX_PNET_ID_LEN];
+	u8 mlen = 0;
+
+	/* check if peer is smc capable */
+	if (!ntp->syn_smc) {
+		new_smc->use_tcp = 1;
+		goto out_connected;
+	}
+
+	kernel_getpeername(newtcpsocket, (struct sockaddr *)&peeraddr, &len);
+	new_smc->new_lgr = 0;
+	/* do inband token exchange
+	 * receive SMC Proposal CLC message (peer ID, GID, MAC, VLAN)
+	 */
+	rc = clc_wait_msg(new_smc, SMC_CLC_BLOCK, (char *)&pclc_buf,
+			  sizeof(pclc_buf), SMC_CLC_PROPOSAL, &reason_code);
+	if (rc)
+		goto out_err;
+	if (reason_code)
+			goto decline;
+
+	smc_find_roce_resources(&roce_sugg, pnet_id, &peeraddr,
+				newtcpsocket->sk);
+	if (!roce_sugg.ibdev) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+
+	/* is it sufficient to use IP from connection itself? */
+	rc = smc_netinfo_by_tcpsk(&subnet, &mlen, newtcpsocket);
+	pclc2 = (struct smc_proposal_clc_msg2 *)
+		((char *)(pclc) + sizeof(struct smc_proposal_clc_msg)
+	       + pclc->iparea_offset);
+	if ((pclc2->outgoing_subnet != subnet) ||
+	    (pclc2->subnet_mask_bits != mlen)) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+
+	/* allocate connection/link group*/
+	rc = smc_create_conn(new_smc, peeraddr.sin_addr.s_addr,
+			     SMC_SERV, &roce_sugg, &pclc->hdr2, 0);
+	if (rc == -ENOLINK)
+		goto out_err;
+	if (rc) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	lgr = new_smc->conn.lgr;
+	link = &new_smc->conn.lgr->lnk[new_smc->conn.lnk_idx];
+
+	if (lgr->lgr_type == NONE) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	rc = smc_create_rmbs(new_smc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	atomic_set(&new_smc->conn.tx_buf_space, new_smc->conn.rmb_tx_size);
+
+	/* Announce RMB Rtoken */
+	if (new_smc->new_lgr) {
+		rc = smc_initial_map_rmb(new_smc);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_RIM;
+			goto decline;
+		}
+	} else {
+		rc = smc_announce_rmb(new_smc, &reason_code);
+		if (rc)
+			goto decline;
+	}
+	strncpy(lgr->pnet_id, pnet_id, sizeof(pnet_id));
+	rc = smc_send_accept(new_smc, &cclc, &reason_code);
+	if (rc)
+		goto out_err;
+	if (reason_code)
+		goto decline;
+
+	smc_save_peer_info(new_smc, &cclc);
+	if (new_smc->new_lgr) {
+		ntoh_three(&link->qp_num_peer, cclc.qp_num);
+		smc_save_link_info(link, &cclc);
+	}
+	rc = smc_rtoken_handling(lgr, new_smc,
+				 (struct smc_acc_conf_clc_msg *)&cclc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_INT;
+		goto decline;
+	}
+
+	if (new_smc->new_lgr) {
+		rc = smc_ready_link(link);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_INT;
+			goto decline;
+		}
+		lgr->max_links = SMC_MAX_SYM_LINKS;
+		lgr->lgr_type = SINGLE;
+		rc = smc_serv_conf_first_lnk(new_smc, &roce_sugg, &reason_code);
+		if ((rc == -EOPNOTSUPP) || (rc == -EINTR))
+			goto out_err; /* peer is not aware of a problem */
+		if (rc)
+			goto decline;
+		rc = smc_serv_add_lnk(new_smc, &reason_code);
+		if (rc == -ENOLINK)
+			goto out_connected;
+		else if (rc < 0)
+			goto out_err;
+	}
+
+out_connected:
+	if (lgr && mutex_is_locked(&lgr->conf_mutex))
+		mutex_unlock(&lgr->conf_mutex);
+	smc_sock_proc_create(newsmcsk);
+	sk_refcnt_debug_inc(newsmcsk);
+	if (!new_smc->use_tcp)
+		setup_timer(&newsmcsk->sk_timer, smc_keepalive_timer,
+			    (unsigned long)newsmcsk);
+	smc_init_after_clc_handshake(new_smc);
+enqueue:
+	if (lsmc->sk.sk_state == SMC_LISTEN) {
+		smc_accept_enqueue(&lsmc->sk, newsmcsk);
+	} else {
+		smc_destruct_non_accepted(newsmcsk);
+		new_smc = NULL;
+	}
+
+	/* Wake up accept */
+	lsmc->sk.sk_data_ready(&lsmc->sk);
+	sock_put(&lsmc->sk);
+	return;
+
+decline:
+	if (!smc_switch_to_tcp(new_smc, reason_code))
+		goto out_connected;
+
+out_err:
+	if (lgr && mutex_is_locked(&lgr->conf_mutex))
+		mutex_unlock(&lgr->conf_mutex);
+	smc_free_conn(new_smc);
+
+	if (!newsmcsk->sk_err)
+		newsmcsk->sk_err = EPROTO;
+	newsmcsk->sk_state = SMC_CLOSED;
+	if (newsmcsk->sk_err == ETIMEDOUT) {
+		/* forget the new socket */
+		if (new_smc->proc) {
+			proc_remove(new_smc->proc->proc);
+			new_smc->proc = NULL;
+		}
+		sock_set_flag(newsmcsk, SOCK_ZAPPED);
+		sock_set_flag(newsmcsk, SOCK_DEAD);
+		sock_release(newtcpsocket);
+		new_smc->tcpsocket = NULL;
+		sock_put(newsmcsk);
+		sock_put(&lsmc->sk);
+	} else {
+		goto enqueue;
+	}
+}
+
+static void smc_tcp_listen_worker(struct work_struct *work)
+{
+	struct smc_sock *lsmc = container_of(work, struct smc_sock,
+					tcp_listen_work);
+	struct sock *newsmcsk;
+	struct smc_sock *new_smc = NULL;
+	struct socket *newtcpsocket = NULL;
+	struct tcp_sock *ntp;
+	int rc = 0;
+
+	while (lsmc->sk.sk_state == SMC_LISTEN) {
+		rc = kernel_accept(lsmc->tcpsocket, &newtcpsocket, 0);
+		if (lsmc->sk.sk_state == SMC_CLOSED) {
+			if (newtcpsocket)
+				sock_release(newtcpsocket);
+			goto out;
+		}
+		if (rc) {
+			/* find out if there is an error we have to
+			 * restart tcp accept
+			 */
+			lsmc->sk.sk_err = -rc;
+			goto out;
+		}
+		ntp = tcp_sk(newtcpsocket->sk);
+
+		newsmcsk = smc_sock_alloc(NULL, PF_SMC, GFP_KERNEL);
+		if (!newsmcsk) {
+			lsmc->sk.sk_err = ENOMEM;
+			sock_release(newtcpsocket);
+			goto out;
+		}
+
+		new_smc = smc_sk(newsmcsk);
+		new_smc->tcpsocket = newtcpsocket;
+		new_smc->listen_smc = lsmc;
+		sock_hold(&lsmc->sk); /* sock_put in smc_listen_worker */
+		INIT_WORK(&new_smc->listen_work, smc_listen_worker);
+		smc_copy_sockopt_settings(newsmcsk, lsmc->tcpsocket->sk);
+		schedule_work(&new_smc->listen_work);
+	}
+
+out:
+	lsmc->sk.sk_data_ready(&lsmc->sk); /* wake up accept socket */
+}
+
+static int smc_sock_getname(struct socket *sock, struct sockaddr *addr,
+			    int *len, int peer)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc;
+
+	if (smc->tcpsocket)
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket, addr, len,
+						  peer);
+	else
+		rc = -EBADF;
+	return rc;
+}
+
+static int smc_sock_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int rc;
+
+	lock_sock(sk);
+	if (sk->sk_shutdown & SEND_SHUTDOWN) {
+		rc = -EPIPE;
+		goto out_pipe;
+	}
+
+	if (smc->use_tcp) {
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->sendmsg(smc->tcpsocket, msg,
+							  len);
+		else
+			rc = -EBADF;
+		goto out;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ECONNRESET;
+		goto out;
+	}
+	if (smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ENOTCONN;
+		goto out;
+	}
+	if (sk->sk_err) {
+		rc = -EPIPE;
+		goto out_pipe;
+	}
+
+	rc = smc_conn_send(smc, msg, len);
+out_pipe:
+	rc = sk_stream_error(sk, msg->msg_flags, rc);
+out:
+	release_sock(sk);
+
+	return rc;
+}
+
+static int smc_sock_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
+			    int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	lock_sock(sk);
+	if (smc->use_tcp) {
+		if (!smc->tcpsocket)
+			return -ENOTCONN;
+		rc = smc->tcpsocket->ops->recvmsg(smc->tcpsocket, msg, len,
+						  flags);
+	} else {
+		rc = smc_conn_recv(smc, msg, len, flags);
+	}
+	release_sock(sk);
+	return rc;
+}
+
+static unsigned int smc_accept_poll(struct sock *parent)
+{
+	struct smc_sock *isk;
+	struct sock *sk;
+
+	lock_sock(parent);
+	list_for_each_entry(isk, &smc_sk(parent)->accept_q, accept_q) {
+		sk = (struct sock *)isk;
+
+		if (sk->sk_state == SMC_ACTIVE) {
+			release_sock(parent);
+			return POLLIN | POLLRDNORM;
+		}
+	}
+	release_sock(parent);
+
+	return 0;
+}
+
+static unsigned int smc_sock_poll(struct file *file, struct socket *sock,
+				  poll_table *wait)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	unsigned int mask = 0;
+	int rc = 0;
+
+	if ((sk->sk_state == SMC_INIT) ||
+	    (smc->use_tcp && sk->sk_state != SMC_LISTEN)) {
+		if (smc->tcpsocket) {
+			mask = smc->tcpsocket->ops->poll(file, smc->tcpsocket,
+							 wait);
+			if ((sk->sk_state == SMC_INIT) && (mask & POLLOUT)) {
+				sk->sk_err = smc->tcpsocket->sk->sk_err;
+				if (sk->sk_err) {
+					mask |= POLLERR;
+				} else {
+					rc = smc_connect(smc);
+					if (rc)
+						mask |= POLLERR;
+					else
+						mask |= POLLOUT | POLLWRNORM;
+				}
+			}
+		} else {
+			sk->sk_err = EBADF;
+			mask |= POLLERR;
+		}
+	} else {
+		sock_poll_wait(file, sk_sleep(sk), wait);
+		if (sk->sk_state == SMC_LISTEN)
+			mask |= smc_accept_poll(sk);
+		if (sk->sk_err)
+			mask |= POLLERR;
+		if ((sk->sk_shutdown == SHUTDOWN_MASK) ||
+		    (sk->sk_state == SMC_CLOSED))
+			mask |= POLLHUP;
+		if (sk->sk_shutdown & RCV_SHUTDOWN)
+			mask |= POLLIN | POLLRDNORM | POLLRDHUP;
+		if (smc_to_read(smc))
+			mask |= POLLIN | POLLRDNORM;
+		if (sk->sk_state == SMC_APPLCLW1)
+			mask |= POLLIN;
+		if (!(sk->sk_shutdown & SEND_SHUTDOWN)) {
+			if (atomic_read(&smc->conn.tx_buf_space)) {
+				mask |= POLLOUT | POLLWRNORM;
+			} else {
+				set_bit(SOCK_ASYNC_NOSPACE,
+					&sk->sk_socket->flags);
+				set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+			}
+		} else {
+			mask |= POLLOUT | POLLWRNORM;
+		}
+		if ((smc->conn.rx_urg_state & SMC_URG_VALID) &&
+		    !(smc->conn.rx_urg_state & (SMC_URG_READ | SMC_URG_RECV)))
+			mask |= POLLPRI;
+	}
+	return mask;
+}
+
+static int smc_conn_shutdown(struct smc_sock *smc, int how)
+{
+	struct sock *sk = &smc->sk;
+	int rc = 0, ret = 0, old_state;
+
+	if (how == SHUT_RD)
+		return rc;
+
+	lock_sock(sk);
+	old_state = sk->sk_state;
+
+	if (how == SHUT_RDWR) {
+		smc_sk_release(smc, 1);
+		goto out;
+	}
+
+	/* how = SHUT_WR */
+	switch (sk->sk_state) {
+	/* Normal termination - Actice close parts */
+	case SMC_ACTIVE:
+		sk->sk_state = SMC_PEERCLW1;
+		/* fall through */
+	case SMC_PEERCLW1:
+		smc_wait_close(smc);
+		/* fall through */
+	case SMC_PEERCLW2:
+		/* if shutdown(both) and unread data then connection reset
+		 * if outstanding data to be written or RDMA not compl,
+		 * then mask conn as pending close
+		 */
+		smc->conn.local_tx_ctrl.conn_state_flags.sending_done = 1;
+		ret = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		break;
+	/* Normal termination - Passive close parts */
+	case SMC_APPLCLW1:
+		smc_wait_close(smc);
+		sk->sk_state = SMC_APPLCLW2;
+		smc->conn.local_tx_ctrl.conn_state_flags.sending_done = 1;
+		ret = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		break;
+	default:
+		break;
+	}
+
+out:
+	if (old_state != sk->sk_state)
+		sk->sk_state_change(&smc->sk);
+	release_sock(sk);
+	if (ret)
+		rc = -ENOTCONN;
+	return rc;
+}
+
+static int smc_sock_shutdown(struct socket *sock, int how)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (how > SHUT_RDWR)
+		return -EINVAL;
+
+	kernel_sock_shutdown(smc->tcpsocket, how);
+
+	if (!smc->use_tcp)
+		rc = smc_conn_shutdown(smc, how);
+	sk->sk_shutdown |= ++how;
+	return rc;
+}
+
+static u32 smc_keepalive_time_elapsed(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+
+	lgr = smc->conn.lgr;
+	link = &lgr->lnk[smc->conn.lnk_idx];
+	return (u32)jiffies - link->rcv_tstamp;
+}
+
+static void smc_set_keepalive(struct sock *sk, int valbool)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct sock *tp = smc->tcpsocket->sk;
+	u32 ka_tim;
+
+	tcp_set_keepalive(tp, valbool);
+	if (valbool)
+		sock_set_flag(tp, SOCK_KEEPOPEN);
+	else
+		sock_reset_flag(tp, SOCK_KEEPOPEN);
+	if (smc->use_tcp)
+		return;
+	if (sk->sk_state == SMC_ACTIVE) {
+		ka_tim = tcp_sk(tp)->keepalive_time ? : TCP_KEEPALIVE_TIME;
+		if (valbool && !sock_flag(sk, SOCK_KEEPOPEN)) {
+			sk_reset_timer(sk, &sk->sk_timer,
+				       jiffies + (unsigned long)ka_tim);
+		} else if (!valbool) {
+			sk_stop_timer(sk, &sk->sk_timer);
+			cancel_work_sync(&smc->timer_work);
+		}
+	}
+	if (valbool)
+		sock_set_flag(sk, SOCK_KEEPOPEN);
+	else
+		sock_reset_flag(sk, SOCK_KEEPOPEN);
+}
+
+static int smc_sock_setsockopt(struct socket *sock, int level, int optname,
+			       char __user *optval, unsigned int optlen)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct timer_list *cork_timer = &smc->cork_timer;
+	int val;
+	int valbool;
+	u32 ka_tim, elapsed = 0;
+	int rc = 0;
+
+	if (!smc->tcpsocket)
+		return -EBADF;
+	lock_sock(sk);
+	if (level != SOL_TCP)
+		goto fallback;
+	switch (optname) {
+	case SMC_KEEPALIVE:
+		if (get_user(val, (int __user *)optval)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		valbool = val ? 1 : 0;
+		smc_set_keepalive(sk, valbool);
+		break;
+	case TCP_KEEPIDLE:
+		rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket, SOL_TCP,
+			optname, optval, optlen);
+		if (rc || smc->use_tcp)
+			goto ret;
+		if (get_user(val, (int __user *)optval)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		ka_tim = val;
+		if (sk->sk_state == SMC_ACTIVE) {
+			if (ka_tim && sock_flag(sk, SOCK_KEEPOPEN)) {
+				elapsed = smc_keepalive_time_elapsed(smc);
+				if (ka_tim > elapsed)
+					elapsed = ka_tim - elapsed;
+				sk_reset_timer(sk, &sk->sk_timer,
+					       jiffies + elapsed);
+			} else {
+				sk_stop_timer(sk, &sk->sk_timer);
+				cancel_work_sync(&smc->timer_work);
+			}
+		}
+		break;
+	case TCP_CORK:
+		if (optlen < sizeof(int)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		if (get_user(val, (int __user *)optval)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			smc->tmp_sockopts.cork = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_CORK;
+		} else {
+			rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket,
+					level, optname, optval, optlen);
+			if (!rc && !smc->use_tcp) {
+				if (!val && timer_pending(cork_timer)) {
+					del_timer_sync(cork_timer);
+					destroy_timer_on_stack(cork_timer);
+					smc_write_data(smc);
+				}
+			}
+		}
+		break;
+	case TCP_DEFER_ACCEPT:
+		if (optlen < sizeof(int)) {
+			rc = -EINVAL;
+			goto ret;
+		}
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			if (get_user(val, (int __user *)optval)) {
+				rc = -EFAULT;
+				goto ret;
+			}
+			smc->tmp_sockopts.defer_accept = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_DEFER_ACCEPT;
+		} else {
+			goto fallback;
+		}
+		break;
+	case TCP_NODELAY:
+		if (optlen < sizeof(int)) {
+			rc = -EINVAL;
+			goto ret;
+		}
+		if (get_user(val, (int __user *)optval)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			smc->tmp_sockopts.nodelay = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_NODELAY;
+		} else {
+			if (!smc->use_tcp) {
+				if (val && timer_pending(cork_timer)) {
+					del_timer_sync(cork_timer);
+					destroy_timer_on_stack(cork_timer);
+					smc_write_data(smc);
+				}
+			}
+			goto fallback;
+		}
+		break;
+	default:
+		goto fallback;
+	}
+	goto ret;
+fallback:
+	rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket, level,
+		optname, optval, optlen);
+ret:
+	release_sock(sk);
+	return rc;
+}
+
+static int smc_sock_getsockopt(struct socket *sock, int level, int optname,
+			       char __user *optval, int __user *optlen)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int val, len;
+	int rc = 0;
+
+	if (smc->use_tcp)
+		if (!smc->tcpsocket)
+			return -EBADF;
+		return smc->tcpsocket->ops->getsockopt(smc->tcpsocket, level,
+						       optname, optval, optlen);
+	if (get_user(len, optlen))
+		return -EFAULT;
+	if (len < 0)
+		return -EINVAL;
+
+	if (level == SOL_IP) {
+		switch (optname) {
+		case IP_MTU:
+			if (sock->state != SS_CONNECTED || !smc->conn.lgr)
+				return -ENOTCONN;
+			val = ib_mtu_enum_to_int(
+				smc->conn.lgr->lnk[smc->conn.lnk_idx].mtu_peer);
+				break;
+		default:
+			goto fallback;
+		}
+	} else {
+		if (level != SOL_TCP)
+			goto fallback;
+	}
+	/* level is SOL_TCP */
+	switch (optname) {
+	case TCP_CORK:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = !!smc->tmp_sockopts.cork;
+		else
+			goto fallback;
+		break;
+	case TCP_DEFER_ACCEPT:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = smc->tmp_sockopts.defer_accept;
+		else
+			goto fallback;
+		break;
+	case TCP_NODELAY:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = !!smc->tmp_sockopts.nodelay;
+		else
+			goto fallback;
+		break;
+	default:
+		goto fallback;
+	}
+
+	if (len < sizeof(val) && len > 0 && val >= 0 && val <= 255) {
+		unsigned char ucval = (unsigned char)val;
+
+		len = 1;
+		if (put_user(len, optlen))
+			return -EFAULT;
+		if (copy_to_user(optval, &ucval, 1))
+			return -EFAULT;
+	} else {
+		len = min_t(unsigned int, sizeof(int), len);
+		if (put_user(len, optlen))
+			return -EFAULT;
+		if (copy_to_user(optval, &val, len))
+			return -EFAULT;
+	}
+
+	return rc;
+
+fallback:
+	return smc->tcpsocket->ops->getsockopt(smc->tcpsocket, level,
+					       optname, optval, optlen);
+}
+
+static int smc_sock_ioctl(struct socket *sock, unsigned int cmd,
+			  unsigned long arg)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int answ;
+	struct smc_curs urg, c_curs;
+
+	if (smc->use_tcp) {
+		if (!smc->tcpsocket)
+			return -EBADF;
+		return smc->tcpsocket->ops->ioctl(smc->tcpsocket, cmd, arg);
+	}
+
+	switch (cmd) {
+	case SIOCINQ:
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		if (sk->sk_state == SMC_INIT)
+			answ = 0;
+		else
+			answ = smc_to_read(smc);
+		break;
+	case SIOCATMARK:
+		if ((sk->sk_state == SMC_INIT) ||
+		    (sk->sk_state == SMC_LISTEN)) {
+			answ = 0;
+		} else {
+			struct smc_e_ctrl *tx_ctrl = &conn->local_tx_ctrl;
+
+			urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+			if (!sock_flag(&smc->sk, SOCK_URGINLINE))
+				smc_curs_add(conn->rmb_rx_size, &urg.s.curs, 1);
+
+			c_curs.s.lcurs =
+				atomic64_read(&tx_ctrl->c_curs.s.acurs);
+			if ((conn->rx_urg_state & SMC_URG_VALID) &&
+			    (urg.s.curs.c == c_curs.s.curs.c) &&
+			    (urg.s.curs.w == c_curs.s.curs.w))
+				answ = 1;
+			else
+				answ = 0;
+		}
+		break;
+	case SIOCOUTQ:
+		/* output queue size (not send + not acked) */
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		if (sk->sk_state == SMC_INIT)
+			answ = 0;
+		else
+			answ = smc->conn.rmb_tx_size -
+					atomic_read(&smc->conn.tx_buf_space);
+		break;
+	case SIOCOUTQNSD:
+		/* output queue size (not send only) for smc always 0 */
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		answ = 0;
+		break;
+	default:
+		return -ENOIOCTLCMD;
+	}
+
+	return put_user(answ, (int __user *)arg);
+}
+
+static ssize_t smc_sock_sendpage(struct socket *sock, struct page *page,
+				 int offset, size_t size, int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (smc->use_tcp)
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->sendpage(smc->tcpsocket, page,
+							   offset, size, flags);
+		else
+			rc = -EBADF;
+	else
+		rc = sock_no_sendpage(sock, page, offset, size, flags);
+
+	return rc;
+}
+
+static ssize_t smc_sock_splice_read(struct socket *sock, loff_t *ppos,
+				    struct pipe_inode_info *pipe, size_t len,
+				    unsigned int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (smc->use_tcp) {
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->splice_read(smc->tcpsocket,
+							      ppos, pipe, len,
+							      flags);
+		else
+			rc = -EBADF;
+	} else {
+		if (*ppos)
+			/* we don't support offsets on our socket */
+			return -ESPIPE;
+		lock_sock(sk);
+		rc = smc_conn_splice_read(smc, pipe, len, flags);
+		release_sock(sk);
+	}
+
+	return rc;
+}
+
+/* must look like tcp */
+static const struct proto_ops smc_sock_ops = {
+	.family		= PF_SMC,
+	.owner		= THIS_MODULE,
+	.release	= smc_sock_release,
+	.bind		= smc_sock_bind,
+	.connect	= smc_sock_connect,
+	.socketpair	= sock_no_socketpair,
+	.accept		= smc_sock_accept,
+	.getname	= smc_sock_getname,
+	.poll		= smc_sock_poll,
+	.ioctl		= smc_sock_ioctl,
+	.listen		= smc_sock_listen,
+	.shutdown	= smc_sock_shutdown,
+	.setsockopt	= smc_sock_setsockopt,
+	.getsockopt	= smc_sock_getsockopt,
+	.sendmsg	= smc_sock_sendmsg,
+	.recvmsg	= smc_sock_recvmsg,
+	.mmap		= sock_no_mmap,
+	.sendpage	= smc_sock_sendpage,
+	.splice_read	= smc_sock_splice_read,
+};
+
+static int smc_sock_init_tcp(struct smc_sock *smc)
+{
+	int rc;
+
+	rc = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
+			      &smc->tcpsocket);
+	smc->sk.sk_rcvbuf = max_t(size_t, smc_def_rcvbuf_size,
+					  smc->tcpsocket->sk->sk_rcvbuf);
+	smc->sk.sk_sndbuf = max_t(size_t, smc_def_sndbuf_size,
+					  smc->tcpsocket->sk->sk_sndbuf);
+	tcp_sk(smc->tcpsocket->sk)->nonagle |= TCP_NAGLE_OFF | TCP_NAGLE_PUSH;
+	return rc;
+}
+
+static void smc_timer_worker(struct work_struct *);
+
+static void smc_free_proc_worker(struct work_struct *work)
+{
+	struct smc_proc_dir_entry *proc_to_free =
+		container_of(work, struct smc_proc_dir_entry, free_proc_work);
+	proc_remove(proc_to_free->proc);
+	kfree(proc_to_free);
+}
+
+static void smc_sock_destruct(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+
+	if (sk->sk_state != SMC_CLOSED)
+		return;
+	sk->sk_state = SMC_DESTRUCT;
+	if (!sock_flag(sk, SOCK_DEAD))
+		return;
+
+	if (smc->proc) {
+		smc->proc->sk = NULL;
+		schedule_work(&smc->proc->free_proc_work);
+		smc->proc = NULL;
+	}
+
+	smc_free_conn(smc);
+
+	sk_refcnt_debug_dec(sk);
+}
+
+static void smc_release_worker(struct work_struct *work)
+{
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+					release_work);
+
+	lock_sock(&smc->sk); /* make sure smc_sock_release is done */
+	if (!smc->use_tcp)
+		smc_wait_for_pending_sends(smc);
+	release_sock(&smc->sk);
+	sk_stop_timer(&smc->sk, &smc->sk.sk_timer);
+	cancel_work_sync(&smc->timer_work);
+	del_timer_sync(&smc->fin_timer);
+	destroy_timer_on_stack(&smc->fin_timer);
+	sock_put(&smc->sk);
+}
+
+static void smc_timer_worker(struct work_struct *work)
+{
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+					timer_work);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct llc_qentry *qentry;
+	int i, probes, rest_time, lnk_idx;
+
+	lgr = smc->conn.lgr;
+	probes = tp->keepalive_probes ? : TCP_KEEPALIVE_PROBES;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		link = &lgr->lnk[i];
+		lnk_idx = link - &lgr->lnk[0];
+		if (atomic_read(&link->state) != SMC_LINK_UP)
+			continue;
+		/* initiate keepalive */
+		rest_time = llc_initiate(lgr, LLC_GRP_TEST_LINK);
+		if (!rest_time) {
+			link->probes_out++;
+			if ((link->probes_out >= probes) &&
+			    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0)) {
+				schedule_work(&lgr->link_down_work);
+			}
+			return;
+		}
+		if (llc_send_test_link(link, "KEEPALIVE_CHECK")) {
+			llc_stop(lgr, LLC_GRP_TEST_LINK);
+			return;
+		}
+		qentry = llc_wait(lgr, LLC_TESTLINK, LLC_WAIT_TIMER_TIMEO,
+				  LLC_TEST_LINK);
+		if (!qentry) {
+			link->probes_out++;
+			if ((link->probes_out >= probes) &&
+			    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0)) {
+				schedule_work(&lgr->link_down_work);
+			}
+		} else {
+			link->probes_out = 0;
+		}
+		kfree(qentry);
+		llc_stop(lgr, LLC_GRP_TEST_LINK);
+	}
+}
+
+static void smc_keepalive_timer(unsigned long data)
+{	/* derived from tcp_keepalive_timer() */
+	struct sock *sk = (struct sock *)data;
+	struct smc_sock *smc = smc_sk(sk);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct inet_connection_sock *icsk = inet_csk(smc->tcpsocket->sk);
+	u32 elapsed, time, intvl;
+
+	/* Only process if socket is not in use. */
+	bh_lock_sock(sk);
+	if (sock_owned_by_user(sk)) {
+		/* Try again later */
+		sk_reset_timer(sk, &sk->sk_timer, jiffies + (HZ / 20));
+		goto out;
+	}
+
+	if (!sock_flag(sk, SOCK_KEEPOPEN) || (sk->sk_state != SMC_ACTIVE))
+		goto out;
+
+	time = tp->keepalive_time ? : TCP_KEEPALIVE_TIME;
+	if (!time)
+		goto out;
+	intvl = tp->keepalive_intvl ? : TCP_KEEPALIVE_INTVL;
+	elapsed = smc_keepalive_time_elapsed(smc);
+
+	if ((elapsed >= intvl) &&
+	    (!icsk->icsk_user_timeout ||
+	     (elapsed >= icsk->icsk_user_timeout))) {
+		schedule_work(&smc->timer_work);
+		elapsed = intvl;
+	}
+
+	sk_reset_timer(sk, &sk->sk_timer, jiffies + (unsigned long)elapsed);
+out:
+	bh_unlock_sock(sk);
+	sock_put(sk); /* taken in sk_reset_timer() */
+}
+
+static void smc_corked_write_cb(unsigned long arg)
+{
+	struct smc_sock *smc = (struct smc_sock *)arg;
+
+	schedule_delayed_work(&smc->write_work, 0);
+}
+
+static void smc_fin_cb(unsigned long arg)
+{
+	struct smc_sock *smc = (struct smc_sock *)arg;
+	struct sock *tcpsk;
+
+	sock_hold(&smc->sk);
+	if (smc->sk.sk_state == SMC_CLOSED)
+		goto out;
+	smc->sk.sk_err = ETIMEDOUT;
+	smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+	if (smc->tcpsocket && smc->tcpsocket->sk) {
+		smc->tcpsocket->sk->sk_err = ECONNABORTED;
+		smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+	}
+	smc->sk.sk_state = SMC_CLOSED;
+	smc->sk.sk_shutdown = smc->sk.sk_shutdown | RCV_SHUTDOWN;
+	if (smc->tcpsocket) {
+		tcpsk = smc->tcpsocket->sk;
+		tcpsk->sk_shutdown = tcpsk->sk_shutdown | RCV_SHUTDOWN;
+	}
+	sock_set_flag(&smc->sk, SOCK_DEAD);
+	smc->sk.sk_state_change(&smc->sk);
+	wake_up_interruptible(&smc->rx_waiter);
+	smc_sock_wake_tx(&smc->sk);
+	smc_sock_wake_rx(&smc->sk);
+	if (cmpxchg(&smc->sock_put_done, 0, 1) == 0)
+		schedule_work(&smc->release_work);
+out:
+	sock_put(&smc->sk);
+}
+
+static struct sock *smc_sock_alloc(struct socket *sock, int proto, gfp_t prio)
+{
+	struct sock *sk;
+	struct smc_proc_dir_entry *pde = NULL;
+	struct smc_sock *smc;
+
+	pde = kzalloc(sizeof(*pde), GFP_KERNEL);
+	if (!pde)
+		return NULL;
+
+	sk = sk_alloc(&init_net, PF_SMC, prio, &smc_proto, 0);
+	if (!sk) {
+		kfree(pde);
+		return NULL;
+	}
+
+	pde->sk = sk;
+	smc = smc_sk(sk);
+	smc->tcpsocket = NULL;
+	smc->use_tcp = 0;
+	memset(&smc->conn, 0, sizeof(struct smc_connection));
+	smc->spd = NULL;
+	smc->proc = pde;
+
+	sock_init_data(sock, sk); /* sets sk_refcnt to 1 */
+	sk->sk_state = SMC_INIT;
+	INIT_LIST_HEAD(&smc->accept_q);
+	spin_lock_init(&smc->accept_q_lock);
+	sk->sk_destruct = smc_sock_destruct;
+	INIT_WORK(&smc->release_work, smc_release_worker);
+	INIT_WORK(&smc->timer_work, smc_timer_worker);
+	INIT_WORK(&smc->proc->free_proc_work, smc_free_proc_worker);
+	sock_reset_flag(sk, SOCK_ZAPPED);
+	sock_reset_flag(sk, SOCK_DONE);
+	sk->sk_protocol = proto;
+	init_waitqueue_head(&smc->splice_waiter);
+	setup_timer(&smc->cork_timer, smc_corked_write_cb, (unsigned long)smc);
+	setup_timer(&smc->fin_timer, smc_fin_cb, (unsigned long)smc);
+	smc_init_tmp_sockopts(smc);
+
+	return sk;
+}
+
+static int smc_sock_create(struct net *net, struct socket *sock, int protocol,
+			   int kern)
+{
+	struct sock *sk;
+	struct smc_sock *smc;
+	int rc = 0;
+
+	sock->state = SS_UNCONNECTED;
+
+	switch (sock->type) {
+	case SOCK_STREAM:
+		sock->ops = &smc_sock_ops;
+		break;
+	default:
+		return -ESOCKTNOSUPPORT;
+	}
+
+	sk = smc_sock_alloc(sock, protocol, GFP_KERNEL);
+	if (!sk)
+		return -ENOMEM;
+
+	smc = smc_sk(sk);
+	rc = smc_sock_init_tcp(smc);
+	if (rc) {
+		sk_free(sk);
+		smc = NULL;
+		goto out;
+	}
+	smc_sock_proc_create(sk);
+	setup_timer(&sk->sk_timer, smc_keepalive_timer, (unsigned long)sk);
+
+	sk_refcnt_debug_inc(sk);
+out:
+	return rc;
+}
+
+static const struct net_proto_family smc_sock_family_ops = {
+	.family	= AF_SMC,
+	.owner	= THIS_MODULE,
+	.create	= smc_sock_create,
+};
+
+static int __init af_smc_init(void)
+{
+	int rc;
+
+#ifdef CONFIG_PROC_FS
+	rc = smc_proc_init();
+	if (rc) {
+		rc = -EFAULT;
+		goto out;
+	}
+#else
+	pr_warn("smc requires /proc file system\n");
+	rc = -ENOENT;
+	goto out;
+#endif /* CONFIG_PROC_FS */
+
+	INIT_LIST_HEAD(&smc_ib_devices);
+	spin_lock_init(&smc_ib_dev_lock);
+	INIT_LIST_HEAD(&smc_lgr_list.list);
+	spin_lock_init(&smc_lgr_list.lock);
+	INIT_LIST_HEAD(&smc_lgr_freeing_list);
+	smc_lgr_list.create_conn_pending = 0;
+
+	rc = proto_register(&smc_proto, 0);
+	if (rc) {
+		pr_warn("smc proto_register fails with %d\n", rc);
+		goto out;
+	}
+
+	rc = sock_register(&smc_sock_family_ops);
+	if (rc) {
+		pr_warn("smc sock_register fails with %d\n", rc);
+		goto out_proto;
+	}
+
+	rc = ib_register_client(&smc_ib_client);
+	if (rc)
+		goto out_sock;
+
+	return 0;
+
+out_sock:
+	sock_unregister(PF_SMC);
+out_proto:
+	proto_unregister(&smc_proto);
+out:
+	return rc;
+}
+
+static void __exit af_smc_exit(void)
+{
+	ib_unregister_client(&smc_ib_client);
+	sock_unregister(PF_SMC);
+	proto_unregister(&smc_proto);
+	smc_proc_exit();
+}
+
+module_init(af_smc_init);
+module_exit(af_smc_exit);
+
+MODULE_AUTHOR("Ursula Braun <ursula.braun@de.ibm.com>");
+MODULE_DESCRIPTION("smc socket address family");
+MODULE_VERSION(VERSION);
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_NETPROTO(PF_SMC);
diff --git a/net/smc/af_smc.h b/net/smc/af_smc.h
new file mode 100644
index 0000000..1391cd8
--- /dev/null
+++ b/net/smc/af_smc.h
@@ -0,0 +1,706 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Definitions for the SMC module
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ */
+
+#include <linux/types.h>
+#include <net/tcp_states.h>
+
+extern struct list_head smc_ib_devices;
+extern struct ib_client smc_ib_client;
+
+struct smc_lgr_list_struct {
+	struct list_head list;
+	spinlock_t lock;	/* protects list of link groups */
+	u32 create_conn_pending;/* delay 2nd connection creation */
+};
+
+extern struct smc_lgr_list_struct smc_lgr_list; /* list of link groups */
+extern struct list_head smc_lgr_freeing_list; /* link groups to be freed */
+extern u32 smc_ctrl_buf_cnt;		/* # of ctrl buffers per link */
+extern u32 smc_max_conn_per_lgr;	/* max. # of connections per lgr */
+extern u32 smc_port_add_delay;		/* port add worker delay */
+extern atomic_t smc_lgr_num;		/* for creation of a unique lgr id */
+extern atomic_t smc_reconfiguring;	/* signal port adding / deleting */
+extern unsigned int smc_def_sndbuf_size;	/* minimum sndbuf size */
+extern unsigned int smc_def_rcvbuf_size;	/* minimum rcvbuf size */
+
+#define SMC_VERSION 0x10			/* first 4 bits */
+#define SMC_LOCAL_PEERID_RESET	"%%%%%%%%"
+
+#define SMC_MAX_PNET_ID_LEN	16
+
+#define SMC_FIRST_CONTACT 0x08			/* bit 4 */
+#define SMC_LINK_GROUP_OUT_OF_SYNCH 0x08	/* bit 4 */
+
+#define SMC_MAX_CQE 32768	/* max. # of completion queue elements */
+#define SMC_MAX_WRE 1024	/* max. # of work requests in flight per QP */
+#define SMC_MAX_POLL_CQE 10	/* max. # of compl. queue elements in 1 poll */
+#define SMC_MAX_SYM_LINKS 2	/* max. # of symmetric links */
+#define SMC_MAX_RMB 255		/* max. # of RMBs/connections per link group */
+#define SMC_MIN_RMBE_SIZE 16384	/* minimum size of an RMBE */
+#define SMC_MAX_PORT_ADD_DELAY 60	/* max. delay for port add (in secs) */
+
+#define SMC_RMBE_SIZE_VALUES	16
+
+#define SMC_LNK_EYE_CATCHER	"SMC_LINK"
+#define SMC_LNK_EYE_CATCH_LEN	8
+
+#define SMC_SIZE_OF_CTRL_DATA	44
+#define SMC_SIZE_OF_CTRL_BUF	sizeof(struct rmb_e_ctrl)
+
+#define SMC_URG_VALID		0x01
+#define SMC_URG_RECV		0x02
+#define SMC_URG_READ		0x04
+#define SMC_URG_MASK		0x03
+
+#define SMC_CLC_V1		0x10
+
+#define SMC_CLC_PROPOSAL	0x01
+#define SMC_CLC_ACCEPTANCE	0x02
+#define SMC_CLC_CONFIRMATION	0x03
+#define SMC_CLC_DECLINE		0x04
+
+#define SMC_CLC_BLOCK		0
+#define SMC_CLC_NONBLOCK	1
+
+#define SMC_QP_TIMEOUT			15 /* 4096 * 2 ** timeout usec */
+#define SMC_MIN_RNR_TIMER		5
+#define SMC_RETRY_CNT			7 /* 7: infinite */
+#define SMC_RNR_RETRY			7 /* 7: infinite */
+#define LLC_WAIT_FIRST_TIMEO		(5 * HZ)
+#define LLC_WAIT_TIMEO			(2 * HZ)
+#define LLC_WAIT_TIMER_TIMEO		HZ
+#define SMC_FREE_LGR_TIMEO		(600 * HZ)
+#define SMC_WAIT_PENDING_SENDS_TIMEO	(LLC_WAIT_TIMEO - 1 * HZ)
+#define SMC_WAIT_FREE_CTRL_BUF_TIMEO	HZ
+#define SMC_WAIT_RX_SPACE_TIMEO		(10 * HZ)
+#define SMC_WAIT_TX_SPACE_TIMEO		(10 * HZ)
+
+#define SMC_CLC_DEC_RUI		0x01000000  /* insufficient resources         */
+#define SMC_CLC_DEC_RIM		0x01010000  /* insufficient memory resources  */
+#define SMC_CLC_DEC_RIQ		0x01020000  /* insufficient QP resources      */
+#define SMC_CLC_DEC_TU		0x02000000  /* timeout                        */
+#define SMC_CLC_DEC_TPC		0x02010000  /* timeout w4 proposal            */
+#define SMC_CLC_DEC_TAS		0x02020000  /* timeout w4 accept              */
+#define SMC_CLC_DEC_TCC		0x02030000  /* timeout w4 confirm             */
+#define SMC_CLC_DEC_TCL		0x02040000  /* timeout w4 QP confirm          */
+#define SMC_CLC_DEC_CUC		0x03000000  /* configuration error            */
+#define SMC_CLC_DEC_CSLV	0x03010000  /* server lacks RoCE acc. to VLAN */
+#define SMC_CLC_DEC_CNIC	0x03020000  /* RNIC error                     */
+#define SMC_CLC_DEC_SU		0x04000000  /* synchronization error          */
+#define SMC_CLC_DEC_SFCB	0x04010000  /* first contact bit expected     */
+#define SMC_CLC_DEC_FORM	0x05000000  /* format error in received resp. */
+#define SMC_CLC_DEC_REPLY	0x06000000  /* reply to a received decline    */
+#define SMC_CLC_DEC_SEND	0x07000000  /* sending problem                */
+#define SMC_CLC_DEC_INT		0x99990000  /* internal error                 */
+
+#define SMC_LLC_DEL_NOLNK	0x00100000  /* Unknown Link ID (no link)      */
+#define SMC_LLC_DEL_NOLGR	0x00200000  /* Unknown Link Group             */
+
+#define SMC_SOCKOPTS_CORK		0x00000001
+#define SMC_SOCKOPTS_DEFER_ACCEPT	0x00000002
+#define SMC_SOCKOPTS_NODELAY		0x00000004
+
+#ifdef CONFIG_X86
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x01 /* cache_align_size 64 */
+#else
+#ifdef CONFIG_PPC
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x02 /* cache_align_size 128 */
+#else
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x03 /* cache_align_size 256 */
+#endif /* CONFIG_PPC */
+#endif /* CONFIG_X86 */
+
+#define smc_sk(__sk) ((struct smc_sock *)__sk)
+
+#define smc_lnk_downing(state) \
+	(atomic_cmpxchg(state, SMC_LINK_UP, SMC_LINK_DOWN) == SMC_LINK_UP)
+
+#define smc_stop_received(conn) \
+	(conn->local_rx_ctrl.conn_state_flags.sending_done || \
+	 conn->local_rx_ctrl.conn_state_flags.abnormal_close || \
+	 conn->local_rx_ctrl.conn_state_flags.closed_conn)
+
+#define smc_close_received(conn) \
+	(conn->local_rx_ctrl.conn_state_flags.abnormal_close || \
+	 conn->local_rx_ctrl.conn_state_flags.closed_conn)
+
+enum smc_role {		/* possible roles of a link group */
+	SMC_CLNT,	/* client */
+	SMC_SERV	/* server */
+};
+
+enum smc_state {	/* possible states of an SMC socket */
+	SMC_ACTIVE	= TCP_ESTABLISHED,	/* (1) connected */
+	SMC_INIT	= TCP_SYN_SENT,		/* (2) */
+	SMC_CLOSED	= TCP_CLOSE,		/* (7) */
+	SMC_LISTEN	= TCP_LISTEN,	/* (10)required for SO_ACCEPTCONN */
+	SMC_PEERCLW1	= 20,
+	SMC_PEERCLW2	= 21,
+	SMC_APPLCLW1	= 22,
+	SMC_APPLCLW2	= 23,
+	SMC_APPLFINCLW	= 24,
+	SMC_PEERFINCLW	= 25,
+	SMC_PEERABORTW	= 26,
+	SMC_PROCESSABORT = 27,
+	SMC_DESTRUCT	= 32
+};
+
+enum smc_link_state {	/* possible states of a link */
+	SMC_LINK_FREED,
+	SMC_LINK_DOWN,
+	SMC_LINK_ACTIVATING,
+	SMC_LINK_UP
+};
+
+enum llc_ctl_ind {	/* possible llc event control range */
+	LLC_CLNT,	/* as in smc_role */
+	LLC_SERV,	/* as in smc_role */
+	LLC_TESTLINK,
+	LLC_NO_NOTIFY
+};
+
+enum llc_msg_group {	/* classifies llc events */
+	LLC_GRP_NONE		= 0,
+	LLC_GRP_ADD_LINK	= 2,
+	LLC_GRP_DEL_LINK	= 4,
+	LLC_GRP_CONF_RKEY	= 6,
+	LLC_GRP_TEST_LINK	= 7
+};
+
+struct llc_ctl {	/* llc event data */
+	struct llc_qentry	*qentry;	/* llc event data */
+	enum llc_msg_group	active;		/* current llc event group */
+	int			lnk_idx;	/* active link for llc event */
+	int			ctrl_elem;	/* used ctrl buffer element */
+	int			wc_status;	/* status of sent llc event */
+};
+
+struct rmb_conn_state_flags {
+#if defined(__BIG_ENDIAN_BITFIELD)
+	u8	sending_done : 1;
+	u8	closed_conn : 1;
+	u8	abnormal_close : 1;
+	u8	reserved : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+	u8      reserved : 5;
+	u8      abnormal_close : 1;
+	u8      closed_conn : 1;
+	u8      sending_done : 1;
+#endif
+
+} __packed;
+
+struct rmb_producer_flags {
+#if defined(__BIG_ENDIAN_BITFIELD)
+	u8	 write_blocked : 1;	/* Writing Blocked, no rx buf space */
+	u8	 urg_data_pending : 1;	/* Urgent Data Pending */
+	u8	 urg_data_present : 1;	/* Urgent Data Present */
+	u8	 cons_curs_upd_req : 1;	/* cursor update requested */
+	u8	 last_msg_valid_req : 1;/* message replay due to failover */
+	u8	 reserved : 3;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+	u8       reserved : 3;
+	u8       last_msg_valid_req : 1;/* message replay due to failover */
+	u8       cons_curs_upd_req : 1; /* cursor update requested */
+	u8       urg_data_present : 1;  /* Urgent Data Present */
+	u8       urg_data_pending : 1;  /* Urgent Data Pending */
+	u8       write_blocked : 1;     /* Writing Blocked, no rx buf space */
+#endif
+} __packed;
+
+/* in host byte order */
+struct smc_cursor {		/* SMC cursor - an offset in an RMBE */
+	u8	reserved0;
+	u8	reserved;
+	u16	w;		/* window wrap sequence number */
+	u32	c;		/* cursor (= offset) part */
+} __aligned(8);
+
+struct smc_curs {		/* overlay for atomic cursor handling */
+	union {
+		struct smc_cursor	curs;
+		atomic64_t		acurs;
+		long long		lcurs;
+	} s;
+} __aligned(8);
+
+/* in host byte order */
+struct smc_e_ctrl {		/* RMBE control information */
+	u8				ctrl_type;	/* type = 0xFE */
+	u8				ctrl_len;	/* length = 44 */
+	u16				ctrl_seq;	/* connection seq # */
+	u32				ctrl_token;	/* alert_token */
+	struct smc_curs			p_curs;		/* producer cursor */
+	struct smc_curs			c_curs;		/* consumer cursor,
+							 * piggy backed ctrl
+							 */
+	struct rmb_producer_flags	p_flags;	/* conn. tx/rx status */
+	struct rmb_conn_state_flags	conn_state_flags; /* peer conn. status*/
+	u8				reserved[18];
+	u8				reserved1[4];	/* fill up to dw */
+} __packed __aligned(8);
+
+/* in network byte order */
+struct rmb_cursor {		/* SMC cursor */
+	u8	reserved0;
+	u8	reserved;
+	__be16	wwrap_seq;
+	__be32	count;
+} __aligned(8);
+
+struct rmb_curs {
+	union {
+		struct	rmb_cursor	curs;
+		atomic64_t		acurs;
+		long long		lcurs;
+	} r;
+} __aligned(8);
+
+/* in network byte order */
+struct rmb_e_ctrl {
+	u8				ctrl_type;
+	u8				ctrl_len;
+	__be16				ctrl_seq;
+	__be32				ctrl_token;
+	struct rmb_curs			p_curs;
+	struct rmb_curs			c_curs;
+	struct rmb_producer_flags	prod_flags;
+	struct rmb_conn_state_flags	conn_state_flags;
+	u8				reserved[18];
+	u8				reserved1[4];
+} __packed __aligned(8);
+
+enum lgr_type {		/* link status of a link group */
+	INIT,
+	SINGLE,			/* local and peer have just 1 ib-device */
+	ASYMMETRIC_LOCAL,	/* local has 1, peer has 2 ib-devices */
+	ASYMMETRIC_PEER,	/* local has 2, peer has 1 ib-device */
+	SYMMETRIC,		/* local and peer have 2 ib-devices */
+	NONE
+};
+
+struct pending_send {	/* control data for a pending send request */
+	struct smc_connection	*conn;		/* socket connection */
+	struct smc_curs		cursor;		/* tx RMBE cursor sent */
+	struct smc_curs		p_cursor;	/* rx RMBE cursor produced */
+	u64			wr_id_send;	/* work request id sent */
+	u16			ctrl_seq;	/* conn. tx sequence # */
+	u8			post_ctl;	/* llc event control index */
+	u8			reserved;
+};
+
+struct tx_sge {		/* scatter gather element for a send request */
+	u32 offset;
+	u32 len;
+};
+
+struct smc_rtoken {			/* address/key of remote RMB */
+	u64		vaddr[SMC_MAX_SYM_LINKS + 1];
+	u32		rkey[SMC_MAX_SYM_LINKS + 1];
+	atomic_t	active;			/* used / unused entry */
+	u8		link_id[SMC_MAX_SYM_LINKS + 1];		/* link ref. */
+	u8		reserved[7 - SMC_MAX_SYM_LINKS];
+};
+
+struct rmb_tx_addrs {			/* tx RMBE info */
+	struct list_head	list;
+	char			*rmb_tx_dma[SMC_MAX_SYM_LINKS + 1]; /* mapped */
+	char			*rmb_tx;	/* tx RMBE address */
+	u32			used;		/* currently used / unused */
+};
+
+struct rmb_rx_addrs {			/* rx RMBE info */
+	struct list_head	list;
+	char			*rmb_rx_dma[SMC_MAX_SYM_LINKS + 1]; /* mapped */
+	char			*rmb_rx;	/* rx RMBE address */
+	struct ib_mr		*mr_rx[SMC_MAX_SYM_LINKS + 1];	/* rkey */
+	u32			used;		/* currently used / unused */
+};
+
+struct smc_roce_defs {			/* ib-device infos for a link */
+	struct smc_ib_device	*ibdev;
+	union ib_gid		gid;
+	unsigned short		vlan;
+	char			mac[6];
+	u8			sgid_idx;
+	u8			port;
+};
+
+/* SMC Link - long living - for more than 1 SMC Connection */
+struct smc_link {
+	char			lnk_eyecatch[8];
+	struct smc_link_group	*lnk_grp;
+	struct ib_pd		*roce_pd;	/* IB protection domain,
+						 * unique for every ROCE QP
+						 */
+	struct ib_qp		*roce_qp;	/* IB queue pair */
+	struct ib_qp_attr	qp_attr;	/* IB queue pair attributes */
+	struct smc_roce_defs	roce_defs;	/* ib-device infos */
+	atomic_t		state;		/* link state */
+	enum ib_mtu		mtu_peer;	/* mtu size of peer */
+	enum ib_mtu		path_mtu;	/* used mtu */
+	u32			qp_num_peer;	/* QP # of peer */
+	char			mac_peer[6];	/* = gid[8:10] || gid[13:15] */
+	char			gid_peer[16];	/* gid of peer */
+	u8			link_id;	/* unique # within link group */
+
+	struct rmb_e_ctrl	*ctrl_buf_tx;	/* for send of WR_SENDs */
+	dma_addr_t		ctrl_dma_tx;	/* mapped addr for ctrl buff */
+	struct ib_send_wr	*send_wr;	/* send work requests area */
+	struct ib_mr		*mr_tx;		/* from ib_get_dma_mr */
+	struct ib_sge		*send_sge;	/* send scatter gather area */
+	struct pending_send	*pending;	/* in-flight sends area */
+	unsigned long		*pending_bit_mask; /* used ctrl_buf_tx elems  */
+	u64			wr_id_send;	/* seq # of last sent WQE */
+	u32			psn_initial;	/* starting send WQE seq # */
+	u32			send_wr_num;	/* # of send ctrl buffers */
+
+	struct rmb_e_ctrl	*ctrl_buf_rx;	/* for recv of WR_SENDs */
+	dma_addr_t		ctrl_dma_rx;	/* mapped addr for ctrl buff */
+	struct ib_recv_wr	*recv_wr;	/* recv work requests area */
+	struct ib_sge		*recv_sge;	/* recv scatter gather area */
+	u64			wr_id_recv;	/* seq # of last recv WQE */
+	u32			psn_peer;	/* starting recv WQE seq # */
+	u32			recv_wr_num;	/* # of recv ctrl buffers */
+	u32			rcv_tstamp;	/* time of last recv event */
+
+	wait_queue_head_t	wqe_waiter;	/* w4 free ctrl_buf_tx elem */
+	unsigned int		probes_out;	/* # of TESTLINK probes */
+};
+
+/* SMC Link Group */
+struct smc_link_group {
+	struct list_head	list;
+	enum smc_role		role;		/* client or server */
+	enum lgr_type		lgr_type;	/* link status for lgr */
+	u32			lgr_id;		/* unique lgr id */
+	__be32			daddr;		/* destination ip address */
+	__be32			subnet;	/* subnet mask */
+	u8			mask_len;	/* # of significant mask bits */
+	unsigned short		vlan;		/* vlan id of lgr */
+	char			pnet_id[SMC_MAX_PNET_ID_LEN];
+						/* pnet_id of ROCEs */
+	char			peer_peer_id[8];/* unique system id of peer */
+	u8			max_links;	/* allowed # of links (2) */
+	struct mutex		conf_mutex;	/* link added or removed */
+	struct rb_root		conns_all;	/* connection tree */
+	rwlock_t		conns_lock;	/* protects conns_all */
+	unsigned int		conns_num;	/* current # of connections */
+	struct list_head	rmb_tx_bufs[16]; /* tx buffers */
+	struct list_head	rmb_rx_bufs[16]; /* rx buffers */
+	struct smc_rtoken	rtok[SMC_MAX_RMB]; /* remote addr/key pairs */
+	struct smc_link		lnk[SMC_MAX_SYM_LINKS + 1]; /* links */
+	struct list_head	llc_event_q;	/* queue for llc events */
+	spinlock_t		llc_event_q_lock; /* protects llc_event_q */
+	struct llc_qentry	*delayed_q;	/* postpone llc event handl. */
+	struct work_struct	llc_event_work; /* llc event worker */
+	struct llc_ctl		llc_ctl[3];	/* llc event controller */
+	spinlock_t		llc_ctl_lock;	/* protects llc_ctl */
+	wait_queue_head_t	llc_waiter;	/* w4 next llc event */
+	struct workqueue_struct	*llc_wq;	/* work queue for llc events */
+	struct delayed_work	free_lgr_work;	/* delayed freeing of an lgr */
+	struct delayed_work	free_lgr_final_work; /* final lgr freeing */
+	struct smc_ib_device	*ibdev_up;	/* ib-device - new link */
+	struct work_struct	link_up_work;	/* adding link to lgr */
+	struct work_struct	link_down_work;	/* removing link from lgr */
+	struct work_struct	llc_add_link_work; /* llc add event worker */
+	struct work_struct	llc_del_link_work; /* llc del event worker */
+	struct work_struct	llc_conf_rkey_work;/* llc confirm rkey worker*/
+	struct work_struct	llc_del_rkey_work; /* llc del rkey worker */
+	atomic_t		link_num;	/* unique id last added link */
+	u32			lnk_up;		/* link up in progress */
+	u32			lnk_down[SMC_MAX_SYM_LINKS + 1];
+						/* link down in progress */
+	int			cache_align_size;	/* rdma_write align */
+	u8			port_up;	/* ib-device port - new link */
+	char			nwm_data[16];	/* network management data */
+	u8			nwm_flags;	/* network management flags */
+};
+
+struct smc_connection {
+	struct rb_node		alert_node;
+	struct smc_link_group	*lgr;		/* link group of connection */
+	u32			alert_token_local; /* created with custom logic
+						    * to address RMB and RMBE
+						    */
+	int			lnk_idx;	/* link currently used */
+
+	int			rtok_idx;	/* idx to peer rkey/addr */
+	int			peer_rx_buf_len; /* without eyecatcher */
+	u8			peer_tcp_conn_idx; /* from tcp handshake */
+
+	struct smc_e_ctrl	local_tx_ctrl;	/* preparation for sending */
+	struct smc_curs		local_tx_ctrl_fin; /* last completed target
+						    * offset
+						    */
+	struct smc_curs		tx_curs_prep;	/* tx - prepared data */
+	struct smc_curs		tx_curs_sent;	/* tx - sent data */
+	struct smc_curs		tx_curs_fin;	/* tx - confirmed by peer */
+	struct smc_curs		tx_urg_curs;	/* position of urgent byte */
+	atomic_t		tx_buf_space;	/* remaining space in tx rmb */
+	u16			tx_seq;		/* sequence # for WR SENDs */
+	u16			tx_seq_fin;	/* sequence # confirmed */
+	spinlock_t		send_lock;	/* protect wr_sends */
+	/* tx buffer */
+	struct rmb_tx_addrs	*rmb_tx_elem;	/* tx RMBE addresses */
+	int			rmb_tx_size;	/* tx RMBE size <== sock wmem */
+	int			rmb_tx_size_short;
+	int			tx_rmb_reused;	/* new / reused RMB */
+
+	struct smc_e_ctrl	local_rx_ctrl;	/* filled during event_handl. */
+	struct smc_curs		rx_curs_confirmed; /* confirmed to peer */
+	struct smc_curs		local_rx_spliced; /* pos of in-flight data */
+	struct smc_curs		rx_urg_curs;	/* cursor to urgent data */
+	u8			rx_urg_data;	/* 1 byte urgent data */
+	u8			rx_urg_state;	/* urg data avail / rcvd */
+	u16			ctrl_seq_rcvd;	/* seq # of last recv */
+	atomic_t		rx_buf_space;	/* remaining space in rx rmb */
+	atomic_t		bytes_to_rcv;	/* arrived data,
+						 * not yet received
+						 */
+	/* rx buffer */
+	struct rmb_rx_addrs	*rmb_rx_elem;
+	int			rmb_rx_size;	/* rx RMBE size <== sock rmem */
+	int			rmb_rx_size_short;
+	int			rx_rmb_reused;	/* new / reused TMB */
+	int			upd_cons_limit;	/* lower limit for consumer
+						 * cursor update
+						 */
+};
+
+struct smc_ib_device {				/* list of ib-devices */
+	struct list_head	list;
+	struct ib_device	*dev;
+	struct ib_device_attr	attr;		/* ib device attributes */
+	struct ib_port_attr	pattr[2];	/* ib dev. port attributes */
+	struct ib_event_handler	event_handler;	/* global ib_event handler */
+	struct ib_cq		*roce_cq_send;	/* send completion queue */
+	struct ib_cq		*roce_cq_recv;	/* recv completion queue */
+	struct tasklet_struct	recv_tasklet;
+	struct tasklet_struct	send_tasklet;
+	struct work_struct	port_err_work;	/* port error worker */
+	struct delayed_work	port_add_work;	/* port add worker */
+	u32			port_err[2];	/* port error triggered */
+	u32			port_add[2];	/* port add triggered */
+};
+
+/* eye catcher "SMCR" EBCDIC */
+static const char SMC_EC[] = {0xe2, 0xd4, 0xc3, 0xd9};
+
+struct smc_clc_msg_hdr {	/* header1 of clc messages */
+	u8 ec_clce[4];		/* eye catcher */
+	u8 type;		/* proposal / accept / confirm / decline */
+	__be16 length;
+	u8 flags;
+} __packed;
+
+struct smc_clc_msg_hdr2 {	/* header2 of clc messages */
+	u8 peer_id[8];		/* unique system id */
+	union ib_gid gid;
+	u8 mac[6];
+} __packed;
+
+struct smc_proposal_clc_msg {	/* proposal clc message */
+	struct smc_clc_msg_hdr hdr;
+	struct smc_clc_msg_hdr2 hdr2;
+	__u16 iparea_offset;	/* offset to IP address information area */
+} __packed;
+
+struct smc_proposal_clc_msg2 {
+	__be32 outgoing_subnet;	/* subnet mask */
+	u8 subnet_mask_bits;	/* number of significant bits in mask */
+	u8 reserved[2];
+	u8 n;			/* number of IPv6 prefixes in prefix array */
+	u8 ec_pclcf[4];		/* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_acc_conf_clc_msg {	/* accept / confirm clc message */
+	struct smc_clc_msg_hdr hdr;
+	struct smc_clc_msg_hdr2 hdr2;
+	u8 qp_num[3];		/* QP number */
+	__be32 rmb_rkey;	/* RMB rkey */
+	u8 tcp_conn_idx;	/* TCP connection index, which RMBE in RMB */
+	__be32 rmb_alert_token; /* unique connection id */
+	u8 flags2;		/* server's RMB buf size (compressed notation)
+				 * + QP mtu
+				 */
+	u8 reserved;
+	__be64 rmb_vaddr;	/* RMB virtual address */
+	u8 reserved2;
+	u8 psn[3];		/* initial packet sequence number */
+	u8 ec_clcf[4]; /* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_decline_clc_msg {
+	struct smc_clc_msg_hdr hdr;
+	u8 peer_id[8];		/* sender peer_id */
+	u32 peer_diagnosis;	/* diagnosis information */
+	u8 reserved2[4];
+	u8 ec_dclcf[4]; /* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_tmp_sockopts {
+	unsigned int	set;	/* indicates what sockopts were user-set */
+	int		cork;
+	int		defer_accept;
+	int		nodelay;
+};
+
+struct smc_proc_dir_entry {
+	struct sock *sk;
+	struct proc_dir_entry *proc;
+	struct work_struct free_proc_work;
+};
+
+struct smc_sock {				/* smc socket */
+	struct sock sk;
+	struct socket *tcpsocket;		/* internal tcp socket */
+	struct smc_connection conn;		/* connection */
+	struct sockaddr *addr;			/* inet connect address */
+	bool new_lgr;				/* new lgr created for conn. */
+	bool use_tcp;				/* fallback to tcp */
+	struct smc_sock *listen_smc;		/* listen parent */
+	struct work_struct tcp_listen_work;	/* handle tcp socket accepts */
+	struct work_struct listen_work;		/* prepare new accept socket */
+	struct list_head	accept_q;	/* sockets to be accepted */
+	spinlock_t		accept_q_lock;	/* protects accept_q */
+	struct work_struct release_work;	/* w4 finished sents before
+						 * closing socket finally
+						 */
+	wait_queue_head_t prep_waiter;		/* w4 prep sent when closing */
+	wait_queue_head_t destruct_waiter;	/* w4 pending sends */
+	struct timer_list	fin_timer;	/* ensure socket closing even
+						 * if peer is gone
+						 */
+	u32 sock_put_done;			/* allow race free closing */
+	struct work_struct timer_work;		/* keepalive TESTLINK sending */
+	struct delayed_work write_work;		/* RDMA writes of tx data */
+	wait_queue_head_t rx_waiter;		/* w4 space in rx RMBE */
+	wait_queue_head_t	splice_waiter;
+	struct splice_pipe_desc *spd;
+	struct timer_list	cork_timer;
+	struct smc_tmp_sockopts tmp_sockopts;
+	struct smc_proc_dir_entry *proc;
+	char proc_name[48];			/* sockname in /proc/net/smc */
+};
+
+int smc_create_conn(struct smc_sock *, __be32, enum smc_role,
+		    struct smc_roce_defs *, struct smc_clc_msg_hdr2 *, u32);
+void smc_find_roce_resources(struct smc_roce_defs *, char *,
+			     struct sockaddr_in *, struct sock *);
+void smc_find_alt_roce_resources(struct smc_roce_defs *,
+				 struct smc_link_group *, struct smc_link *);
+int smc_gid_by_dev(struct smc_roce_defs *);
+int smc_netinfo_by_tcpsk(__be32 *, u8 *, struct socket *);
+int smc_pnet_by_ibdev(u8, char *, struct smc_ib_device *);
+int smc_get_ib_mac(struct smc_roce_defs *);
+int smc_create_link(struct smc_link_group *, int, struct smc_roce_defs *);
+int smc_port_active(struct smc_ib_device *, u8);
+int smc_ready_link(struct smc_link *);
+void smc_link_up(struct smc_link_group *);
+void smc_link_down(struct smc_link_group *, int);
+int smc_find_rtoken_by_link(struct smc_link_group *, int, u32);
+int smc_modify_qp_rts(struct smc_link *);
+int smc_modify_qp_reset(struct smc_link *);
+int clc_wait_msg(struct smc_sock *, int, char *, int, u8, int *);
+int clc_send_decline(struct smc_sock *, u32, u8);
+
+int smc_create_rmbs(struct smc_sock *);
+int smc_del_rmbs(struct smc_sock *);
+int smc_map_rmbs_tx(struct smc_link_group *, int, int,
+		    struct rmb_tx_addrs *);
+int smc_map_rmbs_rx(struct smc_link_group *, int, int,
+		    struct rmb_rx_addrs *);
+long smc_get_dma_mr(struct smc_link_group *, int, struct rmb_rx_addrs *);
+long smc_map_rmbs(struct smc_sock *);
+long smc_map_rmbs_to_link(struct smc_link_group *, int, int);
+int smc_get_rtoken(struct smc_link_group *);
+u8 smc_compress_bufsize(int);
+int smc_uncompress_bufsize(u8);
+
+int smc_conn_send(struct smc_sock *, struct msghdr *, size_t);
+int smc_conn_recv(struct smc_sock *, struct msghdr *, size_t, int);
+int smc_to_read(struct smc_sock *);
+int smc_get_ctrl_buf(struct smc_link *, u32 *);
+int smc_wr_send(struct smc_link *, struct smc_connection *, u64, u32);
+int smc_wait_rx_data(struct smc_sock *, int, bool, long);
+void smc_sock_wake_tx(struct sock *);
+void smc_sock_wake_rx(struct sock *);
+void smc_send_tasklet_fn(unsigned long);
+void smc_recv_tasklet_fn(unsigned long);
+ssize_t smc_conn_splice_read(struct smc_sock *, struct pipe_inode_info *,
+			     size_t, unsigned int);
+void smc_write_worker(struct work_struct *);
+void smc_write_data(struct smc_sock *);
+
+int smc_check_peer_gone(struct smc_link_group *);
+void smc_free_conn(struct smc_sock *);
+void smc_remove_rx_rmb(struct smc_sock *);
+void smc_free_link(struct smc_link_group *, int);
+void smc_free_lgr(struct smc_link_group *);
+int smc_wait_no_pending_sends_on_link(struct smc_link *);
+int smc_ctrl_send(struct smc_connection *, int);
+int smc_send_close(struct smc_sock *);
+void smc_send_close_alt(struct smc_sock *);
+int smc_switch_conns(struct smc_link_group *, int, int);
+void smc_terminate_conn(struct smc_link_group *);
+void smc_conn_term(struct smc_sock *);
+int smc_prepared_sends(struct smc_connection *);
+int smc_pending_sends(struct smc_connection *);
+void smc_clear_pending_sends(struct smc_connection *);
+
+int smc_sock_proc_create(struct sock *);
+void smc_sock_proc_remove(char *);
+int smc_proc_init(void);
+void smc_proc_exit(void);
+void smc_check_dev_attr(struct smc_ib_device *);
+void smc_check_port_attr(struct smc_ib_device *, int);
+void smc_check_qp_attr(struct smc_link *);
+void smc_cq_handler_recv(struct ib_cq *, void *);
+void smc_cq_handler_send(struct ib_cq *, void *);
+void smc_cq_event_handler(struct ib_event *, void *);
+
+static inline void hton_three(u32 host, u8 *net)
+{
+	u32 t;
+
+	t = cpu_to_be32(host);
+	memcpy(net, ((u8 *)&t) + 1, 3);
+}
+
+static inline void ntoh_three(u32 *host, u8 *net)
+{
+	u32 t = 0;
+
+	memcpy(((u8 *)&t) + 1, net, 3);
+	*host = be32_to_cpu(t);
+}
+
+static inline void smc_curs_dec(int size, struct smc_cursor *curs)
+{
+	if (!curs->c)
+		curs->w--;
+	curs->c = (curs->c + size - 1) % size;
+}
+
+static inline void smc_curs_add(int size, struct smc_cursor *curs, int value)
+{
+	curs->w += (curs->c + value) / size;
+	curs->c = (curs->c + value) % size;
+}
+
+/* calculate cursor difference between old and new, where old <= new */
+static inline int smc_curs_diff(unsigned int size, struct smc_curs *old,
+				struct smc_curs *new)
+{
+	if (old->s.curs.w != new->s.curs.w)
+		return max_t(int, 0, ((size - old->s.curs.c) + new->s.curs.c));
+	return max_t(int, 0, (new->s.curs.c - old->s.curs.c));
+}
diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
new file mode 100644
index 0000000..48f9ae1
--- /dev/null
+++ b/net/smc/smc_core.c
@@ -0,0 +1,3297 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Basic Transport Functions exploiting Infiniband API
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              Stefan Raspl <raspl@linux.vnet.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/wait.h>
+#include <linux/kthread.h>
+#include <linux/hardirq.h>
+#include <net/tcp.h>
+#include <net/sock.h>
+#include <linux/types.h>
+#ifdef CONFIG_S390
+#include <asm/debug.h>
+#endif
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <linux/splice.h>
+#include <linux/mm.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+#define LGR_RECONFIGURING 1
+
+/* Find the connection associated with the given alert token in the link group.
+ * Requires @conns_lock.
+ * @token	alert token to search for
+ * @lgr		 link group to search in
+ * Returns connection associated with token if found, NULL otherwise.
+ */
+static inline
+struct smc_connection *smc_find_conn(u32 token, struct smc_link_group *lgr)
+{
+	struct rb_node *node;
+	struct smc_connection *res = NULL;
+
+	node = lgr->conns_all.rb_node;
+	while (node) {
+		struct smc_connection *cur = rb_entry(node,
+					struct smc_connection, alert_node);
+
+		if (cur->alert_token_local > token) {
+			node = node->rb_left;
+		} else {
+			if (cur->alert_token_local < token) {
+				node = node->rb_right;
+			} else {
+				res = cur;
+				break;
+			}
+		}
+	}
+
+	return res;
+}
+
+/* Register connection's alert token in our lookup structure.
+ * Requires @conns_lock.
+ * @smc		connection to register
+ * Returns 0 on success, != otherwise.
+ */
+static void _smc_register_alert_token(struct smc_connection *conn)
+{
+	struct rb_node **link, *parent = NULL;
+	u32 token = conn->alert_token_local;
+
+	link = &conn->lgr->conns_all.rb_node;
+	while (*link) {
+		struct smc_connection *cur = rb_entry(*link,
+					struct smc_connection, alert_node);
+
+		parent = *link;
+		if (cur->alert_token_local > token)
+			link = &parent->rb_left;
+		else
+			link = &parent->rb_right;
+	}
+	/* Put the new node there */
+	rb_link_node(&conn->alert_node, parent, link);
+	rb_insert_color(&conn->alert_node, &conn->lgr->conns_all);
+}
+
+/* Register connection in link group by assigning an alert token
+ * registered in a search tree.
+ * Requires lgr->conns_lock being held.
+ * Note that '0' is a reserved value and not assigned.
+ */
+static void smc_register_in_lgr(struct smc_connection *conn)
+{
+	static atomic_t nexttoken = ATOMIC_INIT(0);
+
+	WARN_ON(conn->alert_token_local);
+	while (!conn->alert_token_local) {
+		conn->alert_token_local = atomic_inc_return(&nexttoken);
+		if (smc_find_conn(conn->alert_token_local, conn->lgr))
+			conn->alert_token_local = 0;
+	}
+	_smc_register_alert_token(conn);
+	conn->lgr->conns_num++;
+}
+
+/* Unregister and reset the alert token of the given connection
+ */
+static void smc_unregister_in_lgr(struct smc_connection *conn)
+{
+	struct smc_link_group *lgr = conn->lgr;
+
+	if (conn->alert_token_local) {
+		write_lock_bh(&lgr->conns_lock);
+		rb_erase(&conn->alert_node, &lgr->conns_all);
+		lgr->conns_num--;
+		write_unlock_bh(&lgr->conns_lock);
+	}
+}
+
+static void smc_sock_wake_rx_for_tx(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct socket *sock = sk->sk_socket;
+
+	if (sock && (sock->state == SS_CONNECTED) &&
+	    atomic_read(&smc->conn.rx_buf_space))
+		wake_up_interruptible(&smc->rx_waiter);
+}
+
+static void smc_free_rmbs(struct smc_connection *conn)
+{
+	int i;
+
+	if (conn->rmb_tx_elem) {
+		i = ror8(conn->rmb_tx_size_short, 4);
+		cmpxchg(&conn->rmb_tx_elem->used, 1, 0);
+	}
+	conn->rmb_tx_elem = NULL;
+	if (conn->rmb_rx_elem) {
+		i = ror8(conn->rmb_rx_size_short, 4);
+		cmpxchg(&conn->rmb_rx_elem->used, 1, 0);
+	}
+	conn->rmb_rx_elem = NULL;
+}
+
+static int smc_find_same_lnk(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (i == lnk_idx)
+			continue;
+		if ((lgr->lnk[i].roce_defs.ibdev ==
+			lgr->lnk[lnk_idx].roce_defs.ibdev) &&
+		    (lgr->lnk[i].roce_defs.port ==
+			lgr->lnk[lnk_idx].roce_defs.port))
+			return i;
+	}
+	return i;
+}
+
+static void smc_unmap_rmbs(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+	struct rmb_tx_addrs *txkp;
+	struct rmb_rx_addrs *rxkp;
+	struct smc_roce_defs *rocdefs = &lgr->lnk[lnk_idx].roce_defs;
+	int reuse_lnk_idx;
+
+	reuse_lnk_idx = smc_find_same_lnk(lgr, lnk_idx);
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(txkp, &lgr->rmb_tx_bufs[i], list) {
+			if (txkp->rmb_tx_dma[lnk_idx] &&
+			    (reuse_lnk_idx > SMC_MAX_SYM_LINKS))
+				ib_dma_unmap_single(rocdefs->ibdev->dev,
+					(dma_addr_t)txkp->rmb_tx_dma[lnk_idx],
+					smc_uncompress_bufsize(rol8(i, 4)),
+					DMA_TO_DEVICE);
+			txkp->rmb_tx_dma[lnk_idx] = NULL;
+		}
+		list_for_each_entry(rxkp, &lgr->rmb_rx_bufs[i], list) {
+			if (rxkp->rmb_rx_dma[lnk_idx] &&
+			    (reuse_lnk_idx > SMC_MAX_SYM_LINKS))
+				ib_dma_unmap_single(rocdefs->ibdev->dev,
+					(dma_addr_t)rxkp->rmb_rx_dma[lnk_idx],
+					smc_uncompress_bufsize(rol8(i, 4)),
+					DMA_FROM_DEVICE);
+			rxkp->rmb_rx_dma[lnk_idx] = NULL;
+			if (rxkp->mr_rx[lnk_idx])
+				ib_dereg_mr(rxkp->mr_rx[lnk_idx]);
+			rxkp->mr_rx[lnk_idx] = NULL;
+		}
+	}
+}
+
+static void smc_clear_rtokens(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		lgr->rtok[i].rkey[lnk_idx] = 0;
+		lgr->rtok[i].vaddr[lnk_idx] = 0;
+		lgr->rtok[i].link_id[lnk_idx] = 0;
+	}
+}
+
+static void smc_free_link_mem(struct smc_link *lnk)
+{
+	kfree(lnk->pending);
+	lnk->pending = NULL;
+	kfree(lnk->pending_bit_mask);
+	lnk->pending_bit_mask = NULL;
+	kfree(lnk->send_sge);
+	lnk->send_sge = NULL;
+	kfree(lnk->recv_sge);
+	lnk->recv_sge = NULL;
+	kfree(lnk->recv_wr);
+	lnk->recv_wr = NULL;
+	kfree(lnk->send_wr);
+	lnk->send_wr = NULL;
+	kfree(lnk->ctrl_buf_tx);
+	lnk->ctrl_buf_tx = NULL;
+	kfree(lnk->ctrl_buf_rx);
+	lnk->ctrl_buf_rx = NULL;
+}
+
+static int smc_modify_qp_init(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_INIT;
+	qp_attr.pkey_index = 0;
+	qp_attr.port_num = link->roce_defs.port;
+	qp_attr.qp_access_flags = IB_ACCESS_LOCAL_WRITE
+				| IB_ACCESS_REMOTE_WRITE;
+	rc = ib_modify_qp(link->roce_qp, &qp_attr,
+			  IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_ACCESS_FLAGS |
+			  IB_QP_PORT);
+	return rc;
+}
+
+static int smc_modify_qp_rtr(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+	enum ib_qp_attr_mask qp_attr_mask =
+		IB_QP_STATE | IB_QP_AV | IB_QP_PATH_MTU | IB_QP_DEST_QPN |
+		IB_QP_RQ_PSN | IB_QP_MAX_DEST_RD_ATOMIC | IB_QP_MIN_RNR_TIMER |
+		IB_QP_SMAC;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RTR;
+	qp_attr.path_mtu = min(link->path_mtu, link->mtu_peer);
+	qp_attr.ah_attr.port_num = link->roce_defs.port;
+	qp_attr.ah_attr.ah_flags = IB_AH_GRH;
+	qp_attr.ah_attr.grh.hop_limit = 1;
+	memcpy(&qp_attr.ah_attr.grh.dgid, link->gid_peer,
+	       sizeof(link->gid_peer));
+	qp_attr.ah_attr.grh.sgid_index = link->roce_defs.sgid_idx;
+	if (link->roce_defs.vlan) {
+		qp_attr.ah_attr.vlan_id = link->roce_defs.vlan;
+		qp_attr.vlan_id = link->roce_defs.vlan;
+		qp_attr_mask |= IB_QP_VID;
+	}
+	memcpy(&qp_attr.ah_attr.dmac, link->mac_peer,
+	       sizeof(link->mac_peer));
+	qp_attr.dest_qp_num = link->qp_num_peer;
+	qp_attr.rq_psn = link->psn_peer; /* starting receive packet seq # */
+	qp_attr.max_dest_rd_atomic = 1;  /* max # of resources for incoming
+					  * requests
+					  */
+	qp_attr.min_rnr_timer = SMC_MIN_RNR_TIMER;
+	memcpy(&qp_attr.smac, link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+
+	rc = ib_modify_qp(link->roce_qp, &qp_attr, qp_attr_mask);
+	return rc;
+}
+
+int smc_modify_qp_rts(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RTS;
+	qp_attr.timeout = SMC_QP_TIMEOUT;	/* local ack timeout */
+	qp_attr.retry_cnt = SMC_RETRY_CNT;	/* retry count */
+	qp_attr.rnr_retry = SMC_RNR_RETRY;	/* RNR retries, 7=infinite */
+	qp_attr.sq_psn = link->psn_initial;	/* starting send packet seq # */
+	qp_attr.max_rd_atomic = 1;	/* # of outstanding RDMA reads and
+					 * atomic ops allowed
+					 */
+	rc = ib_modify_qp(link->roce_qp, &qp_attr,
+			  IB_QP_STATE | IB_QP_TIMEOUT | IB_QP_RETRY_CNT |
+			  IB_QP_SQ_PSN | IB_QP_RNR_RETRY |
+			  IB_QP_MAX_QP_RD_ATOMIC);
+	return rc;
+}
+
+int smc_modify_qp_reset(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RESET;
+	rc = ib_modify_qp(link->roce_qp, &qp_attr, IB_QP_STATE);
+	return rc;
+}
+
+static inline int smc_outstanding_post_recv(struct smc_link *link)
+{
+	int rc = 0;
+	struct ib_recv_wr *bad_recv_wr = NULL;
+	u32 index;
+	u64 wr_id;
+
+	wr_id = link->wr_id_recv++; /* tasklet context, thus not atomic */
+	index = wr_id % link->recv_wr_num;
+	link->recv_wr[index].wr_id = wr_id;
+	rc = ib_post_recv(link->roce_qp, &link->recv_wr[index], &bad_recv_wr);
+	return rc;
+}
+
+int smc_ready_link(struct smc_link *link)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	int rc = 0;
+	u32 i;
+
+	rc = smc_modify_qp_init(link);
+	if (rc)
+		goto out;
+
+	rc = smc_modify_qp_rtr(link);
+	if (rc)
+		goto out;
+	smc_check_qp_attr(link);
+	rc = ib_req_notify_cq(link->roce_defs.ibdev->roce_cq_recv,
+			      IB_CQ_SOLICITED_MASK);
+	if (rc)
+		goto out;
+	for (i = 0; i < link->recv_wr_num; i++)
+		rc = smc_outstanding_post_recv(link);
+	smc_check_qp_attr(link);
+
+	if (lgr->role == SMC_SERV) {
+		rc = smc_modify_qp_rts(link);
+		if (rc)
+			goto out;
+		smc_check_qp_attr(link);
+	}
+out:
+	return rc;
+}
+
+static void smc_free_spd(struct smc_sock *);
+
+/* reset rx_rmb if rkey exchange fails */
+void smc_remove_rx_rmb(struct smc_sock *smc)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct rmb_rx_addrs *rxkp = conn->rmb_rx_elem;
+	int lnk_idx = conn->lnk_idx;
+	struct smc_link *lnk = &conn->lgr->lnk[lnk_idx];
+	struct smc_roce_defs *rocdefs = &lnk->roce_defs;
+
+	if (rxkp->mr_rx[lnk_idx])
+		ib_dereg_mr(rxkp->mr_rx[lnk_idx]);
+	if (rxkp->rmb_rx_dma[lnk_idx])
+		ib_dma_unmap_single(rocdefs->ibdev->dev,
+				    (dma_addr_t)rxkp->rmb_rx_dma[lnk_idx],
+				    conn->rmb_rx_size, DMA_FROM_DEVICE);
+	rxkp->rmb_rx_dma[lnk_idx] = NULL;
+	list_del(&rxkp->list);
+	kfree(rxkp);
+	conn->rmb_rx_elem = NULL;
+}
+
+void smc_free_conn(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int empty;
+
+	if (lgr) {
+		/* can be called from sk_free in atomic context */
+		smc_clear_pending_sends(&smc->conn);
+		smc_free_rmbs(&smc->conn);
+		smc_unregister_in_lgr(&smc->conn);
+		empty = RB_EMPTY_ROOT(&smc->conn.lgr->conns_all);
+		if (empty)
+			schedule_delayed_work(&lgr->free_lgr_work,
+					      SMC_FREE_LGR_TIMEO);
+		smc->conn.lgr = NULL;
+	}
+	if (smc->spd)
+		smc_free_spd(smc);
+}
+
+void smc_free_link(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct smc_link *lnk = &lgr->lnk[lnk_idx];
+	struct smc_ib_device *ibdev = lnk->roce_defs.ibdev;
+	enum smc_link_state old_state;
+
+	old_state = atomic_cmpxchg(&lnk->state, SMC_LINK_DOWN, SMC_LINK_FREED);
+	if (old_state != SMC_LINK_DOWN) {
+		if (old_state == SMC_LINK_FREED) /* link freeing in progress */
+			while (lnk->roce_defs.ibdev)
+				msleep(10000);
+		return;
+	}
+	lnk->link_id = 0;
+	memset(lnk->lnk_eyecatch, 0, SMC_LNK_EYE_CATCH_LEN);
+	memset(lnk->pending_bit_mask, 0,
+	       BITS_TO_LONGS(smc_ctrl_buf_cnt) * sizeof(unsigned long));
+	if (!lnk->roce_defs.ibdev)
+		return;
+	if (lnk->roce_qp) {
+		ib_destroy_qp(lnk->roce_qp);
+		lnk->roce_qp = NULL;
+	}
+	smc_unmap_rmbs(lgr, lnk_idx);
+	memset(&lnk->roce_defs, 0, sizeof(struct smc_roce_defs));
+	lnk->qp_num_peer = 0;
+	smc_clear_rtokens(lgr, lnk_idx);
+	if (lnk->mr_tx) {
+		ib_dereg_mr(lnk->mr_tx);
+		lnk->mr_tx = NULL;
+	}
+	if (lnk->ctrl_dma_rx) {
+		ib_dma_unmap_single(ibdev->dev, lnk->ctrl_dma_rx,
+				(sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+				    DMA_FROM_DEVICE);
+		lnk->ctrl_dma_rx = 0;
+	}
+	if (lnk->ctrl_dma_tx) {
+		ib_dma_unmap_single(ibdev->dev, lnk->ctrl_dma_tx,
+				(sizeof(struct rmb_e_ctrl) * lnk->send_wr_num),
+				    DMA_TO_DEVICE);
+		lnk->ctrl_dma_tx = 0;
+	}
+	if (lnk->roce_pd) {
+		ib_dealloc_pd(lnk->roce_pd);
+		lnk->roce_pd = NULL;
+	}
+}
+
+static int smc_pending_sends_on_link(struct smc_link *link)
+{
+	u32 i;
+
+	i = find_first_bit(link->pending_bit_mask, link->send_wr_num);
+	if (i < link->send_wr_num)
+		return 1;
+	return 0;
+}
+
+int smc_wait_no_pending_sends_on_link(struct smc_link *link)
+{
+	int rc = 1;
+
+	if (smc_pending_sends_on_link(link)) {
+		rc = wait_event_interruptible_timeout(link->wqe_waiter,
+			!smc_pending_sends_on_link(link),
+			SMC_WAIT_PENDING_SENDS_TIMEO);
+	}
+	return rc;
+}
+
+void smc_free_lgr(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry, *q;
+	struct rmb_tx_addrs *entrytx, *tmptx;
+	struct rmb_rx_addrs *entryrx, *tmprx;
+	int i, rc;
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_del_init(&lgr->list); /* remove from smc_lgr_freeing_list */
+	spin_unlock_bh(&smc_lgr_list.lock);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (smc_lnk_downing(&lgr->lnk[i].state))
+			rc = smc_wait_no_pending_sends_on_link(&lgr->lnk[i]);
+		smc_free_link(lgr, i);
+	}
+	cancel_delayed_work_sync(&lgr->free_lgr_work);
+	cancel_work_sync(&lgr->llc_add_link_work);
+	if (lgr->role == SMC_SERV)
+		cancel_work_sync(&lgr->llc_del_link_work);
+	cancel_work_sync(&lgr->llc_conf_rkey_work);
+	cancel_work_sync(&lgr->llc_del_rkey_work);
+	destroy_workqueue(lgr->llc_wq);
+	spin_lock_bh(&lgr->llc_event_q_lock);
+	list_for_each_entry_safe(qentry, q, &lgr->llc_event_q, list) {
+		list_del_init(&qentry->list);
+		kfree(qentry);
+	}
+	spin_unlock_bh(&lgr->llc_event_q_lock);
+	list_for_each_entry_safe(entrytx, tmptx, &lgr->rmb_tx_bufs[i], list) {
+		list_del(&entrytx->list);
+		kfree(entrytx->rmb_tx);
+		kfree(entrytx);
+	}
+	list_for_each_entry_safe(entryrx, tmprx, &lgr->rmb_rx_bufs[i], list) {
+		list_del(&entryrx->list);
+		kfree(entryrx->rmb_rx);
+		kfree(entryrx);
+	}
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		smc_free_link_mem(&lgr->lnk[i]);
+		lgr->lnk[i].lnk_grp = NULL;
+	}
+	kfree(lgr);
+}
+
+static void smc_free_lgr_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  free_lgr_work.work);
+	struct llc_qentry *qentry;
+	int i, rest_time, rc, peer_notified = 0, peer_gone;
+
+	if (list_empty(&lgr->list))
+		peer_gone = -EPIPE;
+	else
+		peer_gone = smc_check_peer_gone(lgr);
+	if ((lgr->role == SMC_CLNT) && !peer_gone)
+		return;		/* server to trigger lgr removal */
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	read_lock_bh(&lgr->conns_lock);
+	rc = RB_EMPTY_ROOT(&lgr->conns_all);
+	read_unlock_bh(&lgr->conns_lock);
+	if (!rc) {
+		spin_unlock_bh(&smc_lgr_list.lock);
+		return;
+	}
+	if (atomic_read(&smc_reconfiguring) ||
+	    mutex_is_locked(&lgr->conf_mutex)) {
+		/* postpone freeing */
+		spin_unlock_bh(&smc_lgr_list.lock);
+		schedule_delayed_work(&lgr->free_lgr_work,
+				      LLC_WAIT_FIRST_TIMEO);
+		return;
+	}
+	list_move_tail(&lgr->list, &smc_lgr_freeing_list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+	if (peer_gone)
+		goto out;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (!smc_lnk_downing(&lgr->lnk[i].state) ||
+		    (lgr->lgr_type == NONE) ||
+		    peer_notified)
+			continue;
+		rest_time = llc_initiate(lgr, LLC_GRP_DEL_LINK);
+		if (!rest_time) {		/* timeout */
+			llc_stop(lgr, LLC_GRP_DEL_LINK);
+			continue;
+		}
+		if (!&lgr->lnk[i].link_id) {	/* link already freed */
+			llc_stop(lgr, LLC_GRP_DEL_LINK);
+			continue;
+		}
+		atomic_set(&lgr->lnk[i].state, SMC_LINK_UP); /* temporarily */
+		rc = llc_send_del_link(&lgr->lnk[i], NULL, LLC_FLAG_DEL_ORDERLY,
+				       LLC_DEL_PROG_INIT_TERM, LLC_SERV);
+		if (rc >= 0) {
+			lgr->llc_ctl[LLC_SERV].qentry = NULL;
+			qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO,
+					  LLC_DEL_LINK);
+			if (qentry)
+				peer_notified = 1;
+		}
+		llc_stop(lgr, LLC_GRP_DEL_LINK);
+		atomic_set(&lgr->lnk[i].state, SMC_LINK_DOWN);
+	}
+out:
+	llc_kill_waiters(lgr);
+	cancel_work_sync(&lgr->llc_event_work);
+	schedule_delayed_work(&lgr->free_lgr_final_work, 2 * TCP_TIMEWAIT_LEN);
+}
+
+static void smc_free_lgr_final_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  free_lgr_final_work.work);
+
+	smc_free_lgr(lgr);
+}
+
+static inline int smc_splice_in_use(const struct smc_sock *smc)
+{
+	return smc->spd != NULL;
+}
+
+static int smc_splice_in_progress(struct smc_sock *smc)
+{
+	struct smc_curs curs;
+	struct smc_curs spl_curs;
+
+	spl_curs.s.lcurs = atomic64_read(&smc->conn.local_rx_spliced.s.acurs);
+	if (spl_curs.s.curs.reserved0) {
+		curs.s.lcurs =
+		       atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+		if (smc_curs_diff(smc->conn.rmb_rx_size, &curs, &spl_curs))
+			return 1;
+	}
+
+	return 0;
+}
+
+static inline int smc_find_pending_send(struct smc_link *link, u64 wr_id)
+{
+	u32 i;
+
+	for (i = 0; i < link->send_wr_num; i++) {
+		if (link->pending[i].wr_id_send == wr_id)
+			return i;
+	}
+	return link->send_wr_num;
+}
+
+static void smc_flag_sock_err(struct smc_sock *smc)
+{
+	int old_state;
+
+	sock_hold(&smc->sk);
+	smc->sk.sk_err = ECONNABORTED;
+	lock_sock(&smc->sk);
+	old_state = smc->sk.sk_state;
+	smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+	if (smc->tcpsocket && smc->tcpsocket->sk) {
+		smc->tcpsocket->sk->sk_err = ECONNABORTED;
+		smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+	}
+	smc_conn_term(smc);
+	if (!sock_flag(&smc->sk, SOCK_DEAD))
+		smc->sk.sk_error_report(&smc->sk);
+	release_sock(&smc->sk);
+	if (old_state != SMC_INIT) {
+		/* forget about data prepared to be sent */
+		atomic64_set(&smc->conn.tx_curs_prep.s.acurs,
+			     atomic64_read(&smc->conn.tx_curs_sent.s.acurs));
+		wake_up_interruptible(&smc->rx_waiter);
+		smc_sock_wake_tx(&smc->sk);
+		smc_sock_wake_rx(&smc->sk);
+		wake_up(&smc->prep_waiter);
+	}
+	sock_put(&smc->sk);
+	if ((old_state != SMC_INIT) &&
+	    (smc->sk.sk_state == SMC_CLOSED) &&
+	    (sock_flag(&smc->sk, SOCK_DEAD)) &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		schedule_work(&smc->release_work);
+	}
+}
+
+void smc_terminate_conn(struct smc_link_group *lgr)
+{
+	struct smc_sock *smc;
+	struct rb_node *node;
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	if (list_empty(&lgr->list)) { /* already terminated ? */
+		spin_unlock_bh(&smc_lgr_list.lock);
+		return;
+	}
+	list_del_init(&lgr->list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+again:
+	read_lock_bh(&lgr->conns_lock);
+	for (node = rb_first(&lgr->conns_all); node; node = rb_next(node)) {
+		smc = container_of(rb_entry(node, struct smc_connection,
+					    alert_node),
+				   struct smc_sock, conn);
+		if ((smc->sk.sk_err != ECONNABORTED) && (!smc->use_tcp)) {
+			read_unlock_bh(&lgr->conns_lock);
+			smc_flag_sock_err(smc);
+			goto again;
+		}
+	}
+	read_unlock_bh(&lgr->conns_lock);
+}
+
+static int smc_write_space(struct smc_sock *);
+
+static int smc_switch_cursor(struct smc_sock *smc, int to_lnk_idx)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_curs cons, fin;
+	int diff;
+	int rc = 0;
+
+	atomic64_set(&conn->tx_curs_sent.s.acurs,
+		     atomic64_read(&conn->tx_curs_fin.s.acurs));
+	fin.s.lcurs = atomic64_read(&conn->local_tx_ctrl_fin.s.acurs);
+	atomic64_set(&conn->local_tx_ctrl.p_curs.s.acurs, fin.s.lcurs);
+	cons.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	if (((cons.s.curs.w > fin.s.curs.w) ||
+	     ((cons.s.curs.w == fin.s.curs.w) &&
+	      (cons.s.curs.c > fin.s.curs.c))) ||
+	    (!cons.s.curs.w && (fin.s.curs.w == 0xffff))) {
+		diff = smc_curs_diff(conn->peer_rx_buf_len, &fin, &cons);
+		smc_curs_add(conn->rmb_tx_size,
+			     &conn->tx_curs_sent.s.curs, diff);
+		smc_curs_add(conn->rmb_tx_size,
+			     &conn->tx_curs_fin.s.curs, diff);
+		smp_mb__before_atomic();
+		atomic_add(diff, &conn->tx_buf_space);
+		smp_mb__after_atomic();
+		smc_curs_add(conn->peer_rx_buf_len,
+			     &conn->local_tx_ctrl.p_curs.s.curs, diff);
+		smc_curs_add(conn->peer_rx_buf_len,
+			     &conn->local_tx_ctrl_fin.s.curs, diff);
+	}
+	atomic_set(&smc->conn.rx_buf_space, smc_write_space(smc));
+	if ((smc->sk.sk_state != SMC_INIT) &&
+	    (smc->sk.sk_state != SMC_CLOSED) &&
+	    (smc->sk.sk_state != SMC_DESTRUCT)) {
+		conn->local_tx_ctrl.p_flags.last_msg_valid_req = 1;
+		rc = smc_ctrl_send(conn, conn->lnk_idx);
+		conn->local_tx_ctrl.p_flags.last_msg_valid_req = 0;
+		schedule_delayed_work(&smc->write_work, 0);
+		smc_sock_wake_rx_for_tx(&smc->sk);
+	}
+	return rc;
+}
+
+int smc_switch_conns(struct smc_link_group *lgr, int lnk_idx, int dev_err)
+{
+	struct smc_sock *smc;
+	struct smc_connection *conn;
+	struct smc_roce_defs *rd_idx = &lgr->lnk[lnk_idx].roce_defs;
+	int to_lnk_idx, i, rc = 0, found = 0;
+	struct rb_node *node;
+
+	wake_up(&lgr->lnk[lnk_idx].wqe_waiter);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		struct smc_roce_defs *rd_i = &lgr->lnk[i].roce_defs;
+
+		if ((atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP) ||
+		    (i == lnk_idx))
+			continue;
+		if (dev_err && (rd_i->ibdev == rd_idx->ibdev) &&
+		    (rd_i->port == rd_idx->port))
+			continue;
+		found = 1;
+		to_lnk_idx = i;
+		break;
+	}
+	if (!found) {
+		lgr->lgr_type = NONE;
+		wake_up_interruptible(&lgr->llc_waiter);
+		smc_terminate_conn(lgr); /* requires lgr_list_lock and
+					  * lgr->conns_lock
+					  */
+		return -ENOENT;
+	}
+
+again:
+	read_lock_bh(&lgr->conns_lock);
+	for (node = rb_first(&lgr->conns_all); node; node = rb_next(node)) {
+		conn = rb_entry(node, struct smc_connection, alert_node);
+		if (conn->lnk_idx != lnk_idx)
+			continue;
+		smc = container_of(conn, struct smc_sock, conn);
+		if ((smc->sk.sk_state == SMC_CLOSED) ||
+		    (smc->sk.sk_state == SMC_PEERCLW2) ||
+		    (smc->sk.sk_state == SMC_PEERFINCLW) ||
+		    (smc->sk.sk_state == SMC_APPLFINCLW) ||
+		    (smc->sk.sk_state == SMC_APPLCLW2) ||
+		    (smc->sk.sk_state == SMC_PROCESSABORT)) {
+			conn->lnk_idx = to_lnk_idx;
+			continue;
+		}
+		sock_hold(&smc->sk);
+		read_unlock_bh(&lgr->conns_lock);
+		lock_sock(&smc->sk);
+		conn->lnk_idx = to_lnk_idx;
+		if (!(atomic64_read(&conn->tx_curs_prep.s.acurs)) &&
+		    !(atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs)) &&
+		    !conn->local_tx_ctrl.conn_state_flags.closed_conn) {
+			release_sock(&smc->sk);
+			sock_put(&smc->sk);
+			goto again;
+		}
+		rc = smc_switch_cursor(smc, to_lnk_idx);
+		release_sock(&smc->sk);
+		sock_put(&smc->sk);
+		if (rc) {
+			smc_link_down(lgr, to_lnk_idx);
+			break;
+		}
+		goto again;
+	}
+	read_unlock_bh(&lgr->conns_lock);
+	return to_lnk_idx;
+}
+
+void smc_cq_event_handler(struct ib_event *evt, void *ptr)
+{
+	return;
+}
+
+static void smc_qp_event_handler(struct ib_event *evt, void *ptr)
+{
+	struct smc_link *link = (struct smc_link *)ptr;
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_roce_defs *rocdefs = &link->roce_defs;
+	struct smc_ib_device *smc_ibdev = (struct smc_ib_device *)evt->device;
+	int lnk_idx;
+
+	lnk_idx = link - &lgr->lnk[0];
+
+	switch (evt->event) {
+	case IB_EVENT_DEVICE_FATAL:	/* 8 */
+		if (cmpxchg(&smc_ibdev->port_err[rocdefs->port - 1], 0, 1) == 0)
+			schedule_work(&rocdefs->ibdev->port_err_work);
+		break;
+	case IB_EVENT_GID_CHANGE:	 /* 17 */
+	case IB_EVENT_PORT_ERR:		 /* 10 */
+	case IB_EVENT_QP_ACCESS_ERR:	 /* 3 */
+		if ((atomic_read(&link->state) == SMC_LINK_UP) &&
+		    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0))
+			schedule_work(&lgr->link_down_work);
+		break;
+	case IB_EVENT_COMM_EST:		 /* 4 */
+	case IB_EVENT_PORT_ACTIVE:	 /* 9 */
+	default:
+		break;
+	}
+}
+
+static inline void smc_send_tasklet_conn(struct smc_sock *smc)
+{
+	if (smc->sk.sk_socket && atomic_read(&smc->conn.tx_buf_space) &&
+	    test_bit(SOCK_NOSPACE, &smc->sk.sk_socket->flags)) {
+		smc_sock_wake_tx(&smc->sk);
+	}
+	if (smc->sk.sk_state != SMC_ACTIVE)
+		wake_up(&smc->destruct_waiter);
+}
+
+static inline void smc_send_tasklet_wc(struct ib_wc *wc)
+{
+	struct smc_link_group *lgr;
+	struct smc_link *link;
+	struct smc_sock *smc;
+	struct pending_send pnd_snd;
+	u32 pnd_snd_idx;
+	int i, diff;
+
+	link = wc->qp->qp_context;
+	if (!link->link_id ||
+	    (memcmp(link->lnk_eyecatch, SMC_LNK_EYE_CATCHER,
+		    SMC_LNK_EYE_CATCH_LEN)))
+		return;
+	pnd_snd_idx = smc_find_pending_send(link, wc->wr_id);
+	if (pnd_snd_idx == link->send_wr_num)
+		return;
+	memcpy(&pnd_snd, &link->pending[pnd_snd_idx],
+	       sizeof(struct pending_send));
+	if (!test_and_clear_bit(pnd_snd_idx, link->pending_bit_mask))
+		return;
+	lgr = link->lnk_grp;
+	if (wc->status) {
+		int lnk_idx = link - &lgr->lnk[0];
+
+		for_each_set_bit(i, link->pending_bit_mask, link->send_wr_num)
+			clear_bit(i, link->pending_bit_mask);
+		if ((wc->wr_id != 1) &&
+		    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0))
+			schedule_work(&lgr->link_down_work);
+	}
+	if (pnd_snd.conn) {
+		smc = container_of(pnd_snd.conn, struct smc_sock, conn);
+		bh_lock_sock(&smc->sk);
+		if ((!wc->status) &&
+		    (link == &lgr->lnk[pnd_snd.conn->lnk_idx])) {
+			diff = smc_curs_diff(pnd_snd.conn->rmb_tx_size,
+					     &pnd_snd.conn->tx_curs_fin,
+					     &pnd_snd.cursor);
+			smp_mb__before_atomic();
+			atomic_add(diff, &pnd_snd.conn->tx_buf_space);
+			smp_mb__after_atomic();
+			atomic64_set(&pnd_snd.conn->tx_curs_fin.s.acurs,
+				     pnd_snd.cursor.s.lcurs);
+			atomic64_set(&pnd_snd.conn->local_tx_ctrl_fin.s.acurs,
+				     pnd_snd.p_cursor.s.lcurs);
+			pnd_snd.conn->tx_seq_fin = pnd_snd.ctrl_seq;
+		}
+		wake_up(&link->wqe_waiter);
+		smc_send_tasklet_conn(smc);
+		bh_unlock_sock(&smc->sk);
+	} else {
+		if (pnd_snd.post_ctl != LLC_NO_NOTIFY) {
+			lgr->llc_ctl[pnd_snd.post_ctl].ctrl_elem =
+							LLC_SEND_POSTED;
+			lgr->llc_ctl[pnd_snd.post_ctl].wc_status = wc->status;
+			wake_up_interruptible(&lgr->llc_waiter);
+		}
+		wake_up(&link->wqe_waiter);
+	}
+}
+
+void smc_send_tasklet_fn(unsigned long data)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)data;
+	struct ib_wc wc[SMC_MAX_POLL_CQE];
+	int i = 0, rc, rc1 = 0;
+	int polled = 0;
+
+again:
+	polled++;
+	do {
+		rc = ib_poll_cq(ibdev->roce_cq_send, SMC_MAX_POLL_CQE, wc);
+		if (polled == 1) {
+			rc1 = ib_req_notify_cq(ibdev->roce_cq_send,
+					       IB_CQ_NEXT_COMP |
+					       IB_CQ_REPORT_MISSED_EVENTS);
+		}
+		if (!rc)
+			break;
+		for (i = 0; i < rc; i++)
+			smc_send_tasklet_wc(&wc[i]);
+	} while (rc > 0);
+	if (polled == 1)
+		goto again;
+}
+
+void smc_cq_handler_send(struct ib_cq *ib_cq, void *cq_context)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)cq_context;
+
+	tasklet_schedule(&ibdev->send_tasklet);
+}
+
+static inline
+void smc_save_cursor(struct smc_curs *local, struct rmb_curs *peer)
+{
+	struct smc_curs temp, old;
+
+	old.s.lcurs = atomic64_read(&local->s.acurs);
+	temp.s.lcurs = atomic64_read(&peer->r.acurs);
+	temp.s.curs.c = ntohl(temp.s.curs.c);
+	temp.s.curs.w = ntohs(temp.s.curs.w);
+	if ((old.s.curs.w > temp.s.curs.w) && temp.s.curs.w)
+		return;
+	if ((old.s.curs.w == temp.s.curs.w) &&
+	    (old.s.curs.c > temp.s.curs.c))
+		return;
+	atomic64_set(&local->s.acurs, temp.s.lcurs);
+}
+
+static inline
+void smc_set_cursor(struct smc_curs *local, struct rmb_cursor *peer)
+{
+	struct smc_curs temp;
+
+	temp.s.lcurs = atomic64_read(&local->s.acurs);
+	peer->count = htonl(temp.s.curs.c);
+	peer->wwrap_seq = htons(temp.s.curs.w);
+}
+
+static inline
+void smc_save_ctrl(struct smc_e_ctrl *local, struct rmb_e_ctrl *peer)
+{
+	local->p_flags = peer->prod_flags;
+	local->conn_state_flags = peer->conn_state_flags;
+	smc_save_cursor(&local->p_curs, &peer->p_curs);
+	smc_save_cursor(&local->c_curs, &peer->c_curs);
+	local->ctrl_type = peer->ctrl_type;
+	local->ctrl_len = peer->ctrl_len;
+	local->ctrl_seq = ntohs(peer->ctrl_seq);
+	local->ctrl_token = peer->ctrl_token;
+}
+
+static inline
+void smc_set_ctrl(struct smc_e_ctrl *local, struct rmb_e_ctrl *peer)
+{
+	memcpy((char *)peer, (char *)local, SMC_SIZE_OF_CTRL_DATA);
+	peer->prod_flags = local->p_flags;
+	peer->conn_state_flags = local->conn_state_flags;
+	smc_set_cursor(&local->p_curs, &peer->p_curs.r.curs);
+	smc_set_cursor(&local->c_curs, &peer->c_curs.r.curs);
+	peer->ctrl_seq = htons(local->ctrl_seq);
+	peer->ctrl_token = local->ctrl_token;
+}
+
+static void smc_update_cons_curs(struct smc_sock *, struct smc_link *link);
+
+static int smc_recv_validate(struct smc_sock *smc, struct smc_link *link,
+			     u32 index)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+	u16 tmp;
+
+	/* check that seqnum was seen before */
+	tmp = conn->ctrl_seq_rcvd - ntohs(link->ctrl_buf_rx[index].ctrl_seq);
+	if (tmp > 0x7fff) {
+		/* drop connection */
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+		conn->local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			smc->tcpsocket->sk->sk_err = ECONNABORTED;
+			smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+		}
+		conn->lnk_idx = link - &conn->lgr->lnk[0];
+		smc_conn_term(smc);
+		wake_up_interruptible(&smc->rx_waiter);
+		smc_sock_wake_tx(&smc->sk);
+		smc_sock_wake_rx(&smc->sk);
+		if ((smc->sk.sk_state == SMC_CLOSED) &&
+		    (sock_flag(&smc->sk, SOCK_DEAD)) &&
+		    (cmpxchg(&smc->sock_put_done, 0, 1) == 0))
+			schedule_work(&smc->release_work);
+		rc = -ECONNABORTED;
+	} else {
+		conn->local_rx_ctrl.p_flags.cons_curs_upd_req = 1;
+	}
+	return rc;
+}
+
+static void smc_recv_action(struct smc_sock *smc,
+			    struct smc_link *link, u32 index)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_curs prod, cons_old, prod_old;
+	int diff;
+
+	if (link->ctrl_buf_rx[index].prod_flags.last_msg_valid_req) {
+		if (smc_recv_validate(smc, link, index))
+			return;
+	}
+	prod_old.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons_old.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	smc_save_ctrl(&conn->local_rx_ctrl, &link->ctrl_buf_rx[index]);
+	if (!conn->local_rx_ctrl.p_flags.last_msg_valid_req) {
+		if ((conn->local_rx_ctrl.ctrl_seq -
+			(u16)(conn->ctrl_seq_rcvd + 1)) > 0x7fff)
+			return;
+		conn->ctrl_seq_rcvd = conn->local_rx_ctrl.ctrl_seq;
+	}
+	diff = smc_curs_diff(conn->peer_rx_buf_len, &cons_old,
+			     &conn->local_rx_ctrl.c_curs);
+	smp_mb__before_atomic();
+	atomic_add(diff, &conn->rx_buf_space);
+	smp_mb__after_atomic();
+	diff = smc_curs_diff(conn->rmb_rx_size, &prod_old,
+			     &conn->local_rx_ctrl.p_curs);
+	smp_mb__before_atomic();
+	atomic_add(diff, &conn->bytes_to_rcv);
+	smp_mb__after_atomic();
+
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		smc->sk.sk_err = ECONNRESET;
+	if (smc_stop_received(conn)) {
+		smc->sk.sk_shutdown |= RCV_SHUTDOWN;
+		sock_set_flag(&smc->sk, SOCK_DONE);
+		smc_conn_term(smc);
+		smc_sock_wake_rx(&smc->sk);
+		smc_sock_wake_tx(&smc->sk);
+	}
+
+	/* piggy backed tx info */
+	smc_sock_wake_rx_for_tx(&smc->sk);
+
+	if ((smc->sk.sk_state == SMC_CLOSED) &&
+	    smc_close_received(conn) &&
+	    sock_flag(&smc->sk, SOCK_DEAD) &&
+	    !smc->sk.sk_socket &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		wake_up_interruptible(&smc->rx_waiter);
+		/* socket is closed and peer has confirmed close, now we are
+		 * allowed to clean up our sock/rmbe
+		 */
+		schedule_work(&smc->release_work);
+		return;
+	}
+
+	/* socket connected but not accepted */
+	if (!smc->sk.sk_socket)
+		return;
+
+	/* data available */
+	if (conn->local_rx_ctrl.p_flags.urg_data_present) {
+		prod.s.lcurs =
+			atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+		smc_curs_dec(conn->rmb_rx_size, &prod.s.curs);
+		atomic64_set(&conn->rx_urg_curs.s.acurs, prod.s.lcurs);
+		/* set urg_data to previous byte */
+		conn->rx_urg_data =
+			conn->rmb_rx_elem->rmb_rx[prod.s.curs.c];
+		conn->rx_urg_state = SMC_URG_VALID;
+		sk_send_sigurg(&smc->sk);
+	}
+	if ((conn->local_rx_ctrl.p_flags.write_blocked) ||
+	    (conn->local_rx_ctrl.p_flags.cons_curs_upd_req))
+		smc_update_cons_curs(smc, link);
+	if ((smc_to_read(smc) > 0) ||
+	    conn->local_rx_ctrl.p_flags.urg_data_pending ||
+	    smc_stop_received(conn) ||
+	    smc->sk.sk_shutdown & RCV_SHUTDOWN)
+		smc_sock_wake_rx(&smc->sk);
+}
+
+static inline void smc_recv_conn_update(struct rmb_e_ctrl *ctrl,
+					struct smc_link *link, u64 wr_id)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_connection *connection;
+	struct smc_sock *smc;
+	u32 index;
+
+	/* lookup connection */
+	read_lock_bh(&lgr->conns_lock);
+	connection = smc_find_conn(ctrl->ctrl_token, lgr);
+	if (!connection) {
+		read_unlock_bh(&lgr->conns_lock);
+	} else {
+		smc = container_of(connection, struct smc_sock, conn);
+		if (smc->sk.sk_state != SMC_DESTRUCT) {
+			sock_hold(&smc->sk);
+			read_unlock_bh(&lgr->conns_lock);
+			index = wr_id % link->recv_wr_num;
+			bh_lock_sock(&smc->sk);
+			smc_recv_action(smc, link, index);
+			bh_unlock_sock(&smc->sk);
+			sock_put(&smc->sk); /* no free sk in softirq-context */
+		} else {
+			read_unlock_bh(&lgr->conns_lock);
+		}
+	}
+}
+
+static inline void smc_recv_process_wqs(struct ib_wc wc[], int num)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct rmb_e_ctrl *ctrl;
+	int i, idx;
+	u32 index;
+
+	for (i = 0; i < num; i++) {
+		link = wc[i].qp->qp_context;
+		if (memcmp(link->lnk_eyecatch, SMC_LNK_EYE_CATCHER,
+			   SMC_LNK_EYE_CATCH_LEN))
+			continue;
+		lgr = link->lnk_grp;
+		if ((atomic_read(&link->state) == SMC_LINK_DOWN) ||
+		    (atomic_read(&link->state) == SMC_LINK_FREED))
+			continue;
+		if (wc[i].status == IB_WC_SUCCESS) {
+			index = wc[i].wr_id % link->recv_wr_num;
+			ctrl = (struct rmb_e_ctrl *)&link->ctrl_buf_rx[index];
+			link->rcv_tstamp = (u32)jiffies;
+			if (ctrl->ctrl_type == RMBE_CTRL)
+				smc_recv_conn_update(ctrl, link, wc[i].wr_id);
+			else
+				llc_enqueue(link, (struct llc_msg *)ctrl);
+			smc_outstanding_post_recv(link);
+		} else {
+			/* handle status errors
+			 * terminate all connections of this lgr abnormally
+			 */
+			switch (wc[i].status) {
+			case IB_WC_RETRY_EXC_ERR:
+			case IB_WC_RNR_RETRY_EXC_ERR:
+			case IB_WC_WR_FLUSH_ERR:
+				idx = link - &lgr->lnk[0];
+				if ((atomic_read(&link->state) == SMC_LINK_UP)
+				 && (cmpxchg(&lgr->lnk_down[idx], 0, 1) == 0))
+					schedule_work(&lgr->link_down_work);
+				break;
+			default:
+				smc_outstanding_post_recv(link);
+				break;
+			}
+			continue;
+		}
+	}
+}
+
+void smc_recv_tasklet_fn(unsigned long data)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)data;
+	struct ib_wc wc[SMC_MAX_POLL_CQE];
+	int rc;
+	int polled = 0;
+
+again:
+	polled++;
+	do {
+		memset(&wc, 0, sizeof(wc));
+		rc = ib_poll_cq(ibdev->roce_cq_recv, SMC_MAX_POLL_CQE, wc);
+		if (polled == 1)
+			ib_req_notify_cq(ibdev->roce_cq_recv,
+					 IB_CQ_SOLICITED_MASK
+				       | IB_CQ_REPORT_MISSED_EVENTS);
+		if (!rc)
+			break;
+		smc_recv_process_wqs(&wc[0], rc);
+	} while (rc > 0);
+	if (polled == 1)
+		goto again;
+}
+
+void smc_cq_handler_recv(struct ib_cq *ib_cq, void *cq_context)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)cq_context;
+
+	tasklet_schedule(&ibdev->recv_tasklet);
+}
+
+static int smc_alloc_link_mem(struct smc_link *link)
+{
+	/* allocate link related memory */
+	link->ctrl_buf_rx = (struct rmb_e_ctrl *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct rmb_e_ctrl),
+			GFP_KERNEL);
+	if (!link->ctrl_buf_rx)
+		goto no_mem;
+	link->ctrl_buf_tx = (struct rmb_e_ctrl *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct rmb_e_ctrl),
+			GFP_KERNEL);
+	if (!link->ctrl_buf_tx)
+		goto no_mem_ctrl_buf_rx;
+	link->send_wr = (struct ib_send_wr *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct ib_send_wr),
+			GFP_KERNEL);
+	if (!link->send_wr)
+		goto no_mem_ctrl_buf_tx;
+	link->recv_wr = (struct ib_recv_wr *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct ib_recv_wr),
+			GFP_KERNEL);
+	if (!link->recv_wr)
+		goto no_mem_send_wr;
+	link->send_sge = (struct ib_sge *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct ib_sge),
+			GFP_KERNEL);
+	if (!link->send_sge)
+		goto no_mem_recv_wr;
+	link->recv_sge = (struct ib_sge *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct ib_sge),
+			GFP_KERNEL);
+	if (!link->recv_sge)
+		goto no_mem_send_sge;
+	link->pending_bit_mask = kzalloc(BITS_TO_LONGS(smc_ctrl_buf_cnt) *
+					 sizeof(unsigned long),
+					 GFP_KERNEL);
+	if (!link->pending_bit_mask)
+		goto no_mem_recv_sge;
+	link->pending = (struct pending_send *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct pending_send),
+			GFP_KERNEL);
+	if (!link->pending)
+		goto no_mem_pending_bit_mask;
+	return 0;
+
+no_mem_pending_bit_mask:
+	kfree(link->pending_bit_mask);
+no_mem_recv_sge:
+	kfree(link->recv_sge);
+no_mem_send_sge:
+	kfree(link->send_sge);
+no_mem_recv_wr:
+	kfree(link->recv_wr);
+no_mem_send_wr:
+	kfree(link->send_wr);
+no_mem_ctrl_buf_tx:
+	kfree(link->ctrl_buf_tx);
+no_mem_ctrl_buf_rx:
+	kfree(link->ctrl_buf_rx);
+no_mem:
+	return -ENOMEM;
+}
+
+static void smc_init_sge(struct smc_link *lnk)
+{
+	u32 i;
+
+	for (i = 0; i < lnk->send_wr_num; i++) {
+		lnk->send_wr[i].next = NULL;
+		lnk->send_wr[i].sg_list = &lnk->send_sge[i];
+		lnk->send_wr[i].num_sge = 1;
+		lnk->send_wr[i].opcode = IB_WR_SEND;
+		lnk->send_wr[i].send_flags = IB_SEND_SIGNALED
+					   | IB_SEND_SOLICITED | IB_SEND_INLINE;
+		lnk->send_sge[i].addr =
+				lnk->ctrl_dma_tx + i * SMC_SIZE_OF_CTRL_BUF;
+		lnk->send_sge[i].length = SMC_SIZE_OF_CTRL_DATA;
+		lnk->send_sge[i].lkey = lnk->mr_tx->lkey;
+	}
+	for (i = 0; i < lnk->recv_wr_num; i++) {
+		lnk->recv_wr[i].next = NULL;
+		lnk->recv_wr[i].sg_list = &lnk->recv_sge[i];
+		lnk->recv_wr[i].num_sge = 1;
+		lnk->recv_sge[i].addr =
+				lnk->ctrl_dma_rx + i * SMC_SIZE_OF_CTRL_BUF;
+		lnk->recv_sge[i].length = SMC_SIZE_OF_CTRL_BUF;
+		lnk->recv_sge[i].lkey = lnk->mr_tx->lkey;
+	}
+}
+
+int smc_create_link(struct smc_link_group *lgr, int lnk_idx,
+		    struct smc_roce_defs *rocdefs)
+{
+	struct smc_ib_device *smc_ibdev = rocdefs->ibdev;
+	struct ib_device *ibdev;
+	struct smc_link *lnk = &lgr->lnk[lnk_idx];
+	int i, rc = 0;
+
+	if (list_empty(&smc_ib_devices))
+		return -ENODEV;
+	atomic_set(&lnk->state, SMC_LINK_ACTIVATING);
+	init_waitqueue_head(&lnk->wqe_waiter);
+	ibdev = smc_ibdev->dev;
+	memcpy(&lnk->roce_defs, rocdefs, sizeof(struct smc_roce_defs));
+	lnk->path_mtu = smc_ibdev->pattr[rocdefs->port - 1].active_mtu;
+	while (1) {
+		lnk->link_id = atomic_inc_return(&lgr->link_num);
+		if (!lnk->link_id)
+			continue;
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			if ((i != lnk_idx) &&
+			    (lgr->lnk[i].link_id == lnk->link_id))
+				continue;
+		}
+		break;
+	}
+	/* create PD */
+	lnk->roce_pd = ib_alloc_pd(ibdev);
+	if (IS_ERR(lnk->roce_pd)) {
+		atomic_set(&lnk->state, SMC_LINK_FREED);
+		rc = PTR_ERR(lnk->roce_pd);
+		lnk->roce_pd = NULL;
+		return rc;
+	}
+	/* create QP */
+	{
+	struct ib_qp_init_attr qp_attr = {
+		.event_handler = smc_qp_event_handler,
+		.qp_context = lnk,
+		.send_cq = smc_ibdev->roce_cq_send,
+		.recv_cq = smc_ibdev->roce_cq_recv,
+		.srq = NULL,
+		.cap = {
+			.max_send_wr = 3 * smc_ctrl_buf_cnt,
+				/* include unsolicited rdma_writes as well,
+				 * otherwise mlx4 code reaches overflow
+				 */
+			.max_recv_wr = 10 * smc_ctrl_buf_cnt,
+			.max_send_sge = 4,
+			.max_recv_sge = 1,
+			.max_inline_data = SMC_SIZE_OF_CTRL_BUF,
+		},
+		.sq_sig_type = IB_SIGNAL_REQ_WR,
+		.qp_type = IB_QPT_RC,
+	};
+	lnk->roce_qp = ib_create_qp(lnk->roce_pd, &qp_attr);
+	if (IS_ERR(lnk->roce_qp)) {
+		rc = PTR_ERR(lnk->roce_qp);
+		lnk->roce_qp = NULL;
+		goto dealloc_pd;
+	}
+	}
+	smc_check_qp_attr(lnk);
+	lnk->wr_id_send = 0;
+	lnk->wr_id_recv = 0;
+	lnk->mr_tx = ib_get_dma_mr(lnk->roce_pd, IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(lnk->mr_tx)) {
+		rc = PTR_ERR(lnk->mr_tx);
+		lnk->mr_tx = NULL;
+		goto destroy_qp;
+	}
+	lnk->ctrl_dma_rx = ib_dma_map_single(ibdev, lnk->ctrl_buf_rx,
+				(sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+					     DMA_FROM_DEVICE);
+	if (ib_dma_mapping_error(ibdev, lnk->ctrl_dma_rx)) {
+		lnk->ctrl_dma_rx = 0;
+		rc = -EIO;
+		goto dereg_mr;
+	}
+	lnk->ctrl_dma_tx = ib_dma_map_single(ibdev, lnk->ctrl_buf_tx,
+				sizeof(struct rmb_e_ctrl) * lnk->send_wr_num,
+					     DMA_TO_DEVICE);
+	if (ib_dma_mapping_error(ibdev, lnk->ctrl_dma_tx)) {
+		rc = -EIO;
+		goto dma_unmap;
+	}
+	get_random_bytes_arch(((char *)&lnk->psn_initial) + 1, 3);
+	lnk->rcv_tstamp = 0;
+	lnk->probes_out = 0;
+	smc_init_sge(lnk);
+	memcpy(lnk->lnk_eyecatch, SMC_LNK_EYE_CATCHER, SMC_LNK_EYE_CATCH_LEN);
+	memset(lnk->pending_bit_mask, 0,
+	       BITS_TO_LONGS(smc_ctrl_buf_cnt) * sizeof(unsigned long));
+	return 0;
+
+dma_unmap:
+	ib_dma_unmap_single(ibdev, lnk->ctrl_dma_rx,
+			    (sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+			    DMA_FROM_DEVICE);
+	lnk->ctrl_dma_rx = 0;
+dereg_mr:
+	ib_dereg_mr(lnk->mr_tx);
+	lnk->mr_tx = NULL;
+destroy_qp:
+	ib_destroy_qp(lnk->roce_qp);
+	lnk->roce_qp = NULL;
+dealloc_pd:
+	ib_dealloc_pd(lnk->roce_pd);
+	lnk->roce_pd = NULL;
+	lnk->link_id = 0;
+	memset(&lnk->roce_defs, 0, sizeof(struct smc_roce_defs));
+	atomic_set(&lnk->state, SMC_LINK_FREED);
+	return rc;
+}
+
+static inline
+struct rmb_tx_addrs *smc_get_tx_rmb_slot(struct smc_link_group *lgr,
+					 int start)
+{
+	struct rmb_tx_addrs *slot;
+
+	list_for_each_entry(slot, &lgr->rmb_tx_bufs[start], list) {
+		if (cmpxchg(&slot->used, 0, 1) == 0)
+			return slot;
+	}
+	return NULL;
+}
+
+static inline
+struct rmb_rx_addrs *smc_get_rx_rmb_slot(struct smc_link_group *lgr,
+					 int start)
+{
+	struct rmb_rx_addrs *slot;
+
+	list_for_each_entry(slot, &lgr->rmb_rx_bufs[start], list) {
+		if (cmpxchg(&slot->used, 0, 1) == 0)
+			return slot;
+	}
+	return NULL;
+}
+
+int smc_map_rmbs_tx(struct smc_link_group *lgr, int lnk_idx, int tx_size,
+		    struct rmb_tx_addrs *elem)
+{
+	struct smc_ib_device *dev = lgr->lnk[lnk_idx].roce_defs.ibdev;
+	int rc = 0;
+
+	if (elem->rmb_tx_dma[lnk_idx])
+		return rc; /* already mapped */
+	elem->rmb_tx_dma[lnk_idx] =
+		(char *)ib_dma_map_single(dev->dev, elem->rmb_tx,
+					  tx_size, DMA_TO_DEVICE);
+	if (ib_dma_mapping_error(dev->dev,
+				 (dma_addr_t)elem->rmb_tx_dma[lnk_idx]))
+		rc = -EIO;
+	return rc;
+}
+
+int smc_map_rmbs_rx(struct smc_link_group *lgr, int lnk_idx, int rx_size,
+		    struct rmb_rx_addrs *elem)
+{
+	struct smc_ib_device *dev = lgr->lnk[lnk_idx].roce_defs.ibdev;
+	int rc = 0;
+
+	if (elem->rmb_rx_dma[lnk_idx])
+		return rc; /* already mapped */
+	elem->rmb_rx_dma[lnk_idx] =
+		(char *)ib_dma_map_single(dev->dev, elem->rmb_rx,
+					  rx_size, DMA_FROM_DEVICE);
+	if (ib_dma_mapping_error(dev->dev,
+				 (dma_addr_t)elem->rmb_rx_dma[lnk_idx]))
+		rc = -EIO;
+	return rc;
+}
+
+long smc_get_dma_mr(struct smc_link_group *lgr, int lnk_idx,
+		    struct rmb_rx_addrs *elem)
+{
+	struct ib_mr *mr;
+	long rc = 0;
+
+	if (elem->mr_rx[lnk_idx])
+		return 0; /* already done */
+	/* obtain unique key -
+	 * next invocation of ib_get_dma_mr returns a different key!
+	 */
+	mr = ib_get_dma_mr(lgr->lnk[lnk_idx].roce_pd,
+			   IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE);
+	if (!IS_ERR(mr)) {
+		elem->mr_rx[lnk_idx] = mr;
+	} else {
+		elem->mr_rx[lnk_idx] = NULL;
+		rc = PTR_ERR(mr);
+	}
+	return rc;
+}
+
+long smc_map_rmbs_to_link(struct smc_link_group *lgr, int lnk_idx,
+			  int reuse_lnk_idx)
+{
+	struct rmb_tx_addrs *tx_entry;
+	struct rmb_rx_addrs *rx_entry;
+	long rc = 0;
+	int i;
+
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(tx_entry, &lgr->rmb_tx_bufs[i], list) {
+			if (reuse_lnk_idx <= SMC_MAX_SYM_LINKS) {
+				tx_entry->rmb_tx_dma[lnk_idx] =
+					tx_entry->rmb_tx_dma[reuse_lnk_idx];
+			} else {
+				rc = smc_map_rmbs_tx(lgr, lnk_idx,
+					     smc_uncompress_bufsize(rol8(i, 4)),
+						     tx_entry);
+				if (rc)
+					goto out;
+			}
+		}
+		list_for_each_entry(rx_entry, &lgr->rmb_rx_bufs[i], list) {
+			if (reuse_lnk_idx <= SMC_MAX_SYM_LINKS) {
+				rx_entry->rmb_rx_dma[lnk_idx] =
+					rx_entry->rmb_rx_dma[reuse_lnk_idx];
+			} else {
+				rc = smc_map_rmbs_rx(lgr, lnk_idx,
+					     smc_uncompress_bufsize(rol8(i, 4)),
+						     rx_entry);
+				if (rc)
+					goto out;
+			}
+			rc = smc_get_dma_mr(lgr, lnk_idx, rx_entry);
+			if (rc)
+				goto out;
+		}
+	}
+	return rc;
+
+out:
+	smc_link_down(lgr, lnk_idx);
+	return rc;
+}
+
+long smc_map_rmbs(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	long rc = 0;
+	int i, j;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP)
+			continue;
+		for (j = 0; j < i; j++) {
+			if ((lgr->lnk[i].roce_defs.ibdev ==
+				lgr->lnk[j].roce_defs.ibdev) &&
+			    (lgr->lnk[i].roce_defs.port ==
+				lgr->lnk[j].roce_defs.port))
+				break;
+		}
+		if (!smc->conn.tx_rmb_reused) {
+			if (j == i) {
+				rc = smc_map_rmbs_tx(lgr, i,
+						     smc->conn.rmb_tx_size,
+						     smc->conn.rmb_tx_elem);
+				if (rc) {
+					smc_link_down(lgr, i);
+					continue;
+				}
+			} else {
+				smc->conn.rmb_tx_elem->rmb_tx_dma[i] =
+					smc->conn.rmb_tx_elem->rmb_tx_dma[j];
+			}
+		}
+		if (!smc->conn.rx_rmb_reused) {
+			if (j == i) {
+				rc = smc_map_rmbs_rx(lgr, i,
+						     smc->conn.rmb_rx_size,
+						     smc->conn.rmb_rx_elem);
+				if (rc) {
+					smc_link_down(lgr, i);
+					continue;
+				}
+			} else {
+				smc->conn.rmb_rx_elem->rmb_rx_dma[i] =
+					smc->conn.rmb_rx_elem->rmb_rx_dma[j];
+			}
+			rc = smc_get_dma_mr(lgr, i, smc->conn.rmb_rx_elem);
+			if (rc) {
+				smc_link_down(lgr, i);
+				continue;
+			}
+		}
+	}
+	return rc;
+}
+
+int smc_create_rmbs(struct smc_sock *smc)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_link_group *lgr = conn->lgr;
+	struct rmb_tx_addrs *tx_rmb = NULL;
+	struct rmb_rx_addrs *rx_rmb = NULL;
+	int rmem, wmem, tmp_sl, tmp_ss, sl, ss;
+
+	rmem = smc->sk.sk_rcvbuf;
+	wmem = smc->sk.sk_sndbuf;
+	ss = ror8(smc_compress_bufsize(wmem), 4);
+	sl = smc_uncompress_bufsize(rol8(ss, 4));
+	for (tmp_sl = sl, tmp_ss = ss;
+	     tmp_sl >= SMC_MIN_RMBE_SIZE && tmp_ss >= 0;
+	     tmp_sl = tmp_sl >> 1, tmp_ss--) {
+		tx_rmb = smc_get_tx_rmb_slot(lgr, tmp_ss);
+		if (tx_rmb) {
+			conn->tx_rmb_reused = 1;
+		} else {
+			tx_rmb = (struct rmb_tx_addrs *)
+				kzalloc(sizeof(struct rmb_tx_addrs),
+					GFP_KERNEL);
+			if (!tx_rmb)
+				return -ENOMEM;
+			tx_rmb->rmb_tx = kzalloc(tmp_sl,
+						 GFP_KERNEL | __GFP_NOWARN |
+						 __GFP_NOMEMALLOC |
+						 __GFP_NORETRY);
+			tx_rmb->used = 1;
+			if (tx_rmb->rmb_tx) {
+				list_add(&tx_rmb->list,
+					 &lgr->rmb_tx_bufs[tmp_ss]);
+			}
+		}
+		if (tx_rmb->rmb_tx) {
+			conn->rmb_tx_elem = tx_rmb;
+			memset(conn->rmb_tx_elem->rmb_tx, 0, tmp_sl);
+			break;
+		}
+	}
+	if (!conn->rmb_tx_elem) {
+		kfree(tx_rmb);
+		return -ENOMEM;
+	}
+	conn->rmb_tx_size = tmp_sl;
+	conn->rmb_tx_size_short = rol8(tmp_ss, 4);
+	conn->upd_cons_limit =
+		min_t(int, conn->rmb_rx_size / 10, SOCK_MIN_SNDBUF / 2);
+
+	ss = ror8(smc_compress_bufsize(rmem), 4);
+	sl = smc_uncompress_bufsize(rol8(ss, 4));
+	for (tmp_sl = sl, tmp_ss = ss;
+	     tmp_sl >= SMC_MIN_RMBE_SIZE && tmp_ss >= 0;
+	     tmp_sl = tmp_sl >> 1, tmp_ss--) {
+		rx_rmb = smc_get_rx_rmb_slot(lgr, tmp_ss);
+		if (rx_rmb) {
+			conn->rx_rmb_reused = 1;
+		} else {
+			rx_rmb = (struct rmb_rx_addrs *)
+				kzalloc(sizeof(struct rmb_rx_addrs),
+					GFP_KERNEL);
+			if (!rx_rmb)
+				break;
+			rx_rmb->rmb_rx = kzalloc(tmp_sl,
+						 GFP_KERNEL | __GFP_NOWARN |
+						 __GFP_NOMEMALLOC);
+			rx_rmb->used = 1;
+			if (rx_rmb->rmb_rx)
+				list_add(&rx_rmb->list,
+					 &lgr->rmb_rx_bufs[tmp_ss]);
+		}
+		if (rx_rmb && rx_rmb->rmb_rx) {
+			conn->rmb_rx_elem = rx_rmb;
+			memset(conn->rmb_rx_elem->rmb_rx, 0, tmp_sl);
+			break;
+		}
+	}
+	if (!conn->rmb_rx_elem) {
+		cmpxchg(&tx_rmb->used, 1, 0);
+		conn->rmb_tx_size = 0;
+		conn->rmb_tx_size_short = 0;
+		kfree(rx_rmb);
+		return -ENOMEM;
+	}
+	conn->rmb_rx_size = tmp_sl;
+	conn->rmb_rx_size_short = rol8(tmp_ss, 4);
+
+	return 0;
+}
+
+static struct smc_link *smc_link_match(struct smc_link_group *lgr,
+				       struct smc_clc_msg_hdr2 *hdr2,
+				       u32 qp_num, enum smc_role role,
+				       unsigned long *flags)
+{
+	int i;
+
+	if (mutex_is_locked(&lgr->conf_mutex)) {
+		write_unlock_bh(&lgr->conns_lock);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		mutex_lock(&lgr->conf_mutex);
+		mutex_unlock(&lgr->conf_mutex);
+		spin_lock_bh(&smc_lgr_list.lock);
+		write_lock_bh(&lgr->conns_lock);
+		return (struct smc_link *)LGR_RECONFIGURING;
+	}
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) &&
+		    !memcmp(lgr->lnk[i].gid_peer, &hdr2->gid, 16) &&
+		    !memcmp(lgr->lnk[i].mac_peer, hdr2->mac, 6) &&
+		    ((role == SMC_SERV) ||
+		     (lgr->lnk[i].qp_num_peer == qp_num)))
+			return &lgr->lnk[i];
+	}
+	return NULL;
+}
+
+static int smc_find_link(struct smc_link_group *lgr, int alt)
+{
+	int first = -ENOENT;
+	int second = -ENOENT;
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			first = i;
+			break;
+		}
+	}
+	if (!alt)
+		return first;
+	second = first;
+	for (i = ++first; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			second = i;
+			break;
+		}
+	}
+	return second;
+}
+
+static void smc_link_up_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr
+		= container_of(work, struct smc_link_group, link_up_work);
+
+	if (lgr->lnk_up) {
+		mutex_lock(&lgr->conf_mutex);
+		lgr->lnk_up = 0;
+		smc_link_up(lgr);
+		mutex_unlock(&lgr->conf_mutex);
+	}
+}
+
+static void smc_link_down_worker(struct work_struct *work)
+{
+	int i;
+
+	struct smc_link_group *lgr
+		= container_of(work, struct smc_link_group, link_down_work);
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk_down[i]) {
+			lgr->lnk_down[i] = 0;
+			if (smc_lnk_downing(&lgr->lnk[i].state)) {
+				llc_cancel_wait(&lgr->lnk[i]);
+				mutex_lock(&lgr->conf_mutex);
+				smc_link_down(lgr, i);
+				mutex_unlock(&lgr->conf_mutex);
+			}
+		}
+	}
+}
+
+static int smc_create_lgr(struct smc_sock *smc, __be32 target,
+			  enum smc_role role, struct smc_clc_msg_hdr2 *hdr2)
+{
+	struct smc_link_group *lgr;
+	int i, j, rc;
+	char wq_name[11];
+
+	/* create a new link group */
+	lgr = kzalloc(sizeof(*lgr), GFP_KERNEL);
+	if (!lgr)
+		return -ENOMEM;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		rc = smc_alloc_link_mem(&lgr->lnk[i]);
+		if (rc) {
+			for (j = 0; j < i; j++)
+				smc_free_link_mem(&lgr->lnk[j]);
+			kfree(lgr);
+			return -ENOMEM;
+		}
+		lgr->lnk[i].lnk_grp = lgr;
+	}
+	for (i = 0; i < 16; i++) {
+		INIT_LIST_HEAD(&lgr->rmb_tx_bufs[i]);
+		INIT_LIST_HEAD(&lgr->rmb_rx_bufs[i]);
+	}
+	memcpy(lgr->peer_peer_id, hdr2->peer_id, 8);
+	lgr->daddr = target;
+	lgr->role = role;
+	lgr->lgr_type = INIT;
+	spin_lock_init(&lgr->llc_ctl_lock);
+	INIT_LIST_HEAD(&lgr->llc_event_q);
+	spin_lock_init(&lgr->llc_event_q_lock);
+	lgr->delayed_q = NULL;
+	init_waitqueue_head(&lgr->llc_waiter);
+	INIT_WORK(&lgr->llc_event_work, llc_event_worker);
+	INIT_DELAYED_WORK(&lgr->free_lgr_work, smc_free_lgr_worker);
+	INIT_DELAYED_WORK(&lgr->free_lgr_final_work, smc_free_lgr_final_worker);
+	INIT_WORK(&lgr->llc_add_link_work, llc_process_add_link);
+	INIT_WORK(&lgr->llc_del_link_work, llc_process_del_link);
+	INIT_WORK(&lgr->llc_conf_rkey_work, llc_process_confirm_rkey);
+	INIT_WORK(&lgr->llc_del_rkey_work, llc_process_delete_rkey);
+	lgr->conns_all = RB_ROOT;
+	rwlock_init(&lgr->conns_lock);
+	lgr->lgr_id = atomic_add_return(256, &smc_lgr_num);
+	smc->conn.lgr = lgr;
+	write_lock_bh(&lgr->conns_lock);
+	smc_register_in_lgr(&smc->conn);
+	write_unlock_bh(&lgr->conns_lock);
+	mutex_init(&lgr->conf_mutex);
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_add_tail(&lgr->list, &smc_lgr_list.list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+	mutex_lock(&lgr->conf_mutex);
+	smc->conn.lnk_idx = 0;
+	snprintf(wq_name, sizeof(wq_name), "smcwq%u", lgr->lgr_id);
+	lgr->llc_wq = create_singlethread_workqueue(wq_name);
+	INIT_WORK(&lgr->link_up_work, smc_link_up_worker);
+	INIT_WORK(&lgr->link_down_work, smc_link_down_worker);
+	return 0;
+}
+
+static int smc_check_lgr_consistency(struct smc_sock *smc)
+{
+	u32 reason;
+	u8 flag;
+
+	if (!smc->conn.lgr && !smc->new_lgr) {
+		/* send decline with flag */
+		reason = SMC_CLC_DEC_SU; /* synchronization error */
+		flag = SMC_LINK_GROUP_OUT_OF_SYNCH;
+		clc_send_decline(smc, reason, flag);
+		smc->sk.sk_err = ENOLINK;
+		return -ENOLINK;
+	}
+	if (smc->conn.lgr && smc->new_lgr) {
+		/* something is wrong with this link group */
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_del_init(&smc->conn.lgr->list);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		smc_unregister_in_lgr(&smc->conn); /* takes conns_lock */
+		/* send decline with flag */
+		reason = SMC_CLC_DEC_SU; /* synchronization error */
+		flag = SMC_LINK_GROUP_OUT_OF_SYNCH;
+		clc_send_decline(smc, reason, flag);
+		/* terminate connections */
+		smc->sk.sk_err = ENOLINK;
+		smc_terminate_conn(smc->conn.lgr);
+		return -ENOLINK;
+	}
+	return 0;
+}
+
+int smc_create_conn(struct smc_sock *smc, __be32 target, enum smc_role role,
+		    struct smc_roce_defs *rocdefs,
+		    struct smc_clc_msg_hdr2 *hdr2,
+		    u32 arrival_qp_num)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_link_group *lgr;
+	struct smc_link *link;
+	unsigned long flags;
+	int alt = 0, rc;
+
+	while (atomic_read(&smc_reconfiguring) ||
+	       (cmpxchg(&smc_lgr_list.create_conn_pending, 0, 1) != 0))
+		msleep(20);
+	spin_lock_bh(&smc_lgr_list.lock);
+again:
+	list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+		write_lock_bh(&lgr->conns_lock);
+		if ((!memcmp(lgr->peer_peer_id, hdr2->peer_id, 8)) &&
+		    (lgr->role == role) &&
+		    (lgr->vlan == rocdefs->vlan) &&
+		    ((role == SMC_CLNT) ||
+		     (lgr->conns_num < smc_max_conn_per_lgr))) {
+			link = smc_link_match(lgr, hdr2, arrival_qp_num, role,
+					      &flags);
+			if (!link) {
+				write_unlock_bh(&lgr->conns_lock);
+				continue;
+			}
+			if (link == (struct smc_link *)LGR_RECONFIGURING) {
+				write_unlock_bh(&lgr->conns_lock);
+				goto again;
+			}
+			conn->lgr = lgr;
+			smc_lgr_list.create_conn_pending = 0;
+			if (role == SMC_SERV) {
+				if (lgr->lgr_type == SYMMETRIC)
+					/* try load balancing */
+					alt = lgr->conns_num % 2;
+				conn->lnk_idx = smc_find_link(lgr, alt);
+			} else {
+				conn->lnk_idx = link - &lgr->lnk[0];
+			}
+			smc_register_in_lgr(conn);
+			write_unlock_bh(&lgr->conns_lock);
+			break;
+		}
+		write_unlock_bh(&lgr->conns_lock);
+	}
+	spin_unlock_bh(&smc_lgr_list.lock);
+	if (role == SMC_CLNT) {
+		rc = smc_check_lgr_consistency(smc);
+		if (rc) {
+			smc_lgr_list.create_conn_pending = 0;
+			return -ENOLINK;
+		}
+	}
+	if (!conn->lgr) {
+		smc->new_lgr = 1;
+		rc = smc_create_lgr(smc, target, role, hdr2);
+		smc_lgr_list.create_conn_pending = 0;
+		if (rc)
+			return rc;
+		lgr = conn->lgr;
+		rc = smc_create_link(lgr, 0, rocdefs);
+		if (rc) {
+			lgr->lgr_type = NONE;
+			mutex_unlock(&lgr->conf_mutex);
+			spin_lock_bh(&smc_lgr_list.lock);
+			list_del(&lgr->list);
+			spin_unlock_bh(&smc_lgr_list.lock);
+			kfree(lgr);
+			return rc;
+		}
+		link = &lgr->lnk[conn->lnk_idx];
+	} else {
+		mutex_lock(&lgr->conf_mutex);
+		smc->new_lgr = 0;
+	}
+	conn->local_tx_ctrl.ctrl_type = RMBE_CTRL;
+	conn->local_tx_ctrl.ctrl_len = SMC_SIZE_OF_CTRL_DATA;
+	conn->tx_seq = 0;
+	spin_lock_init(&conn->send_lock);
+	atomic_set(&conn->bytes_to_rcv, 0);
+	conn->lgr->vlan = rocdefs->vlan;
+	init_waitqueue_head(&smc->rx_waiter);
+	return 0;
+}
+
+u8 smc_compress_bufsize(int n)
+{
+	u8 compressed = 0;
+
+	n = (n - 1) >> 14;
+	compressed = ilog2(n) + 1;
+	if (compressed > SMC_RMBE_SIZE_VALUES)
+		compressed = SMC_RMBE_SIZE_VALUES;
+	compressed = rol8(compressed, 4);
+	return compressed;
+}
+
+int smc_uncompress_bufsize(u8 x)
+{
+	u8 left_bits = ror8(x, 4);
+	u32 size;
+
+	size = 0x00000001 << (((int)left_bits) + 14);
+	return size;
+}
+
+static int smc_rdma_write(struct smc_connection *conn, int target_offset,
+			  int lnk_idx, int num, struct tx_sge sge[])
+{
+	struct smc_link_group *lgr = conn->lgr;
+	struct smc_link *link;
+	struct ib_sge sge_send[4];
+	struct ib_send_wr send_wr;
+	struct ib_send_wr *failed_wr;
+	int i, j, rc;
+	int to = target_offset;
+
+	memset(&send_wr, 0, sizeof(send_wr));
+	memset(&sge_send, 0, 4 * sizeof(struct ib_sge));
+	link = &lgr->lnk[lnk_idx];
+	for (i = 0, j = 0; (i < num) && (j < 4); i++, j++) {
+		if ((to % lgr->cache_align_size) &&
+		    ((to / lgr->cache_align_size)) !=
+		     ((to + sge[i].len - 1) / lgr->cache_align_size)) {
+			int initial = lgr->cache_align_size -
+				      to % lgr->cache_align_size;
+
+			/* split into aligned and unaligned part */
+			sge_send[j].addr =
+				((u64)conn->rmb_tx_elem->rmb_tx_dma[lnk_idx]) +
+				sge[i].offset;
+			sge_send[j].length = initial;
+			sge_send[j].lkey = link->mr_tx->lkey;
+			to += initial;
+			sge[i].len -= initial;
+			sge[i].offset += initial;
+			j++;
+		}
+		sge_send[j].addr = ((u64)conn->rmb_tx_elem->rmb_tx_dma[lnk_idx])
+				 + sge[i].offset;
+		sge_send[j].length = sge[i].len;
+		sge_send[j].lkey = link->mr_tx->lkey;
+	}
+	send_wr.wr_id =
+		(u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	send_wr.opcode = IB_WR_RDMA_WRITE;
+	send_wr.wr.rdma.remote_addr =
+		lgr->rtok[conn->rtok_idx].vaddr[lnk_idx]
+	      + target_offset
+	      + ((conn->peer_tcp_conn_idx - 1) * (conn->peer_rx_buf_len));
+	send_wr.wr.rdma.rkey = lgr->rtok[conn->rtok_idx].rkey[lnk_idx];
+	send_wr.num_sge = j;
+	send_wr.sg_list = sge_send;
+	failed_wr = &send_wr;
+	rc = ib_post_send(link->roce_qp, &send_wr, &failed_wr);
+	if (rc) {
+		struct smc_roce_defs *rocd;
+
+		rocd = &link->roce_defs;
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+			if (cmpxchg(&rocd->ibdev->port_err[rocd->port - 1],
+				    0, 1) == 0)
+				schedule_work(&rocd->ibdev->port_err_work);
+	}
+	return rc;
+}
+
+static inline int smc_get_free_ctrl_buf(struct smc_link *link, u32 *elem)
+{
+	*elem = link->send_wr_num;
+	if ((atomic_read(&link->state) == SMC_LINK_DOWN) ||
+	    (atomic_read(&link->state) == SMC_LINK_FREED)) {
+		return -EPIPE;
+	}
+	for_each_clear_bit(*elem, link->pending_bit_mask, link->send_wr_num) {
+		if (!test_and_set_bit(*elem, link->pending_bit_mask))
+			return 0;
+	}
+	*elem = link->send_wr_num;
+	return -EBUSY;
+}
+
+int smc_get_ctrl_buf(struct smc_link *link, u32 *elem)
+{
+	int rc = 0;
+
+	if (in_softirq()) {
+		rc = smc_get_free_ctrl_buf(link, elem);
+		if (rc)
+			return rc;
+	} else {
+		rc = wait_event_interruptible_timeout(link->wqe_waiter,
+				(smc_get_free_ctrl_buf(link, elem) != -EBUSY),
+				SMC_WAIT_FREE_CTRL_BUF_TIMEO);
+		if (!rc)
+			return -EPIPE;
+		if (rc == -ERESTARTSYS)
+			return -EINTR;
+		if (*elem == link->send_wr_num)
+			return -EPIPE;
+	}
+	return 0;
+}
+
+static inline int smc_add_pending_send(struct smc_link *link,
+				       struct smc_connection *conn, u32 *elem)
+{
+	u64 wr_id;
+	int rc = 0;
+
+	wr_id = (u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	rc = smc_get_ctrl_buf(link, elem);
+	if (rc < 0)
+		return rc;
+	memset(&link->pending[*elem], 0, sizeof(struct pending_send));
+	link->pending[*elem].conn = conn;
+	link->pending[*elem].wr_id_send = wr_id;
+	link->send_wr[*elem].wr_id = wr_id;
+	return 0;
+}
+
+int smc_ctrl_send(struct smc_connection *conn, int lnk_idx)
+{
+	struct smc_link *link;
+	struct rmb_e_ctrl ctrl;
+	int rc = 0;
+	u32 elem;
+
+	link = &conn->lgr->lnk[lnk_idx];
+	rc = smc_add_pending_send(link, conn, &elem);
+	if (rc)
+		return rc;
+	spin_lock_bh(&conn->send_lock);
+	if (conn->local_tx_ctrl.p_flags.last_msg_valid_req) {
+		conn->local_tx_ctrl.ctrl_seq = conn->tx_seq_fin;
+	} else {
+		conn->tx_seq++;
+		conn->local_tx_ctrl.ctrl_seq = conn->tx_seq;
+	}
+	link->pending[elem].cursor.s.curs = conn->tx_curs_sent.s.curs;
+	link->pending[elem].p_cursor.s.curs =
+		conn->local_tx_ctrl.p_curs.s.curs;
+	link->pending[elem].ctrl_seq = conn->tx_seq;
+	smc_set_ctrl(&conn->local_tx_ctrl, &ctrl);
+	rc = smc_wr_send(link, conn, (u64)&ctrl, elem);
+	if (!rc) {
+		atomic64_set(&conn->rx_curs_confirmed.s.acurs,
+			    atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs));
+	} else {
+		test_and_clear_bit(elem, link->pending_bit_mask);
+	}
+	spin_unlock_bh(&conn->send_lock);
+	return rc;
+}
+
+/* Send wr via ib_post_send.
+ * Requires conn->send_lock being held if entered with an smc_connection
+ */
+int smc_wr_send(struct smc_link *link, struct smc_connection *conn, u64 addr,
+		u32 index)
+{
+	struct ib_send_wr *failed_wr;
+	struct smc_roce_defs *rocdefs = &link->roce_defs;
+	struct smc_ib_device *ibdev = rocdefs->ibdev;
+	int rc;
+
+	rc = ib_req_notify_cq(ibdev->roce_cq_send,
+			      IB_CQ_SOLICITED_MASK |
+			      IB_CQ_REPORT_MISSED_EVENTS);
+	memcpy(&link->ctrl_buf_tx[index], (void *)addr, SMC_SIZE_OF_CTRL_DATA);
+	failed_wr = &link->send_wr[index];
+
+	rc = ib_post_send(link->roce_qp, &link->send_wr[index], &failed_wr);
+	if (rc) {
+		if ((atomic_read(&link->state) != SMC_LINK_FREED) &&
+		    (cmpxchg(&ibdev->port_err[rocdefs->port - 1], 0, 1) == 0))
+				/* try failover to other link */
+			schedule_work(&ibdev->port_err_work);
+		if (conn) {
+			if (!conn->local_rx_ctrl.p_flags.write_blocked &&
+			    !conn->local_rx_ctrl.p_flags.cons_curs_upd_req) {
+				/* not invoked in softirq context */
+				spin_unlock_bh(&conn->send_lock);
+				msleep(100); /* delay to enable port_err_work
+					      * to be done
+					      */
+				spin_lock_bh(&conn->send_lock);
+			}
+		} else {
+			msleep(100);
+		}
+		rc = 0;
+	}
+	return rc;
+}
+
+static int smc_write_space(struct smc_sock *smc)
+{
+	struct smc_curs prod;
+	struct smc_curs cons;
+	int buffer_len = smc->conn.peer_rx_buf_len;
+	int space;
+
+	prod.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&smc->conn.local_rx_ctrl.c_curs.s.acurs);
+
+	/* determine rx_buf space */
+	space = buffer_len - smc_curs_diff(buffer_len, &cons, &prod);
+
+	return space;
+}
+
+static int smc_wait_rx_space(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int space, rc = 0;
+
+	space = atomic_read(&conn->rx_buf_space);
+	if (space)
+		goto out;
+
+	release_sock(sk);
+	rc = wait_event_interruptible_timeout(smc->rx_waiter,
+		atomic_read(&conn->rx_buf_space) ||
+		sk->sk_err ||
+		smc_stop_received(conn),
+		SMC_WAIT_RX_SPACE_TIMEO);
+	if ((rc == -ERESTARTSYS) || !rc) {
+		/* interrupted or timeout */
+		smc_flag_sock_err(smc);
+		space = sock_error(sk);
+		lock_sock(sk);
+		goto out;
+	}
+	lock_sock(sk);
+	space = atomic_read(&conn->rx_buf_space);
+out:
+	return space;
+}
+
+static int smc_wait_tx_space(struct smc_sock *smc, int flags)
+{   /* derived from sk_stream_wait_memory */
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT);
+	int rc = 0;
+	DEFINE_WAIT(wait);
+
+	if (timeo == MAX_SCHEDULE_TIMEOUT)
+		timeo = (prandom_u32() % (HZ / 5)) + 2;
+	set_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+	if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN) ||
+	    conn->local_tx_ctrl.conn_state_flags.sending_done) {
+		rc = -EPIPE;
+		goto out;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ECONNRESET;
+		goto out;
+	}
+	if (!timeo) {
+		rc = -EAGAIN;
+		goto out;
+	}
+	if (signal_pending(current)) {
+		rc = -EINTR;
+		goto out;
+	}
+	clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
+	if (atomic_read(&conn->tx_buf_space))
+		goto out;
+	set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+	rc = sk_wait_event(sk, &timeo, sk->sk_err ||
+			   (sk->sk_shutdown & SEND_SHUTDOWN) ||
+			   conn->local_tx_ctrl.conn_state_flags.sending_done ||
+			   smc_close_received(conn) ||
+			   atomic_read(&conn->tx_buf_space));
+	clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+	if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN) ||
+	    conn->local_tx_ctrl.conn_state_flags.sending_done) {
+		rc = -EPIPE;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		rc = -ECONNRESET;
+out:
+	finish_wait(sk_sleep(sk), &wait);
+	return rc;
+}
+
+int smc_prepared_sends(struct smc_connection *conn)
+{
+	struct smc_curs prep;
+	struct smc_curs sent;
+
+	prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	sent.s.lcurs = atomic64_read(&conn->tx_curs_sent.s.acurs);
+	return smc_curs_diff(conn->rmb_tx_size, &sent, &prep);
+}
+
+int smc_pending_sends(struct smc_connection *conn)
+{
+	struct smc_link *lnk;
+	u32 i, j;
+
+	if ((!conn) || (!conn->lgr))
+		return 0;
+
+	for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+		lnk = &conn->lgr->lnk[j];
+		for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+			if (lnk->pending[i].conn == conn)
+				return 1;
+		}
+	}
+	return 0;
+}
+
+void smc_clear_pending_sends(struct smc_connection *conn)
+{
+	struct smc_link *lnk;
+	u32 i, j;
+
+	for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+		lnk = &conn->lgr->lnk[j];
+		for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+			if (lnk->pending[i].conn == conn)
+				clear_bit(i, lnk->pending_bit_mask);
+		}
+	}
+}
+
+static inline int smc_give_up_send(struct smc_sock *smc, int copied)
+{
+	struct smc_connection *conn = &smc->conn;
+
+	if (smc->sk.sk_shutdown & SEND_SHUTDOWN ||
+	    conn->local_tx_ctrl.conn_state_flags.abnormal_close)
+		return -EPIPE;
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		return copied ? copied : -ECONNRESET;
+	if (conn->local_rx_ctrl.conn_state_flags.closed_conn)
+		return copied ? copied : -ECONNRESET;
+	return 0;
+}
+
+int smc_conn_send(struct smc_sock *smc, struct msghdr *msg, size_t len)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct smc_curs prep;
+	unsigned long max_wait_for_tx_space;
+	int copy, copied = 0, to_send = len;
+	int writespace;
+	int tx_top = 0, tx_bot = 0;
+	int rc = 0;
+	int tx_cnt_prep;
+	char *buf;
+
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+	}
+	max_wait_for_tx_space =  SMC_WAIT_TX_SPACE_TIMEO + jiffies;
+again:
+	if ((smc->sk.sk_state != SMC_ACTIVE) &&
+	    (smc->sk.sk_state != SMC_APPLCLW1))
+		return (smc->sk.sk_state == SMC_INIT
+		      ? -ENOTCONN : -ECONNRESET);
+
+	writespace = atomic_read(&conn->tx_buf_space);
+	if (!writespace) {
+		writespace = smc_wait_tx_space(smc, msg->msg_flags);
+		if (writespace < 0)
+			return ((copied && (writespace == -EAGAIN))
+			       ? copied : writespace);
+		if (!writespace) {
+			rc = smc_give_up_send(smc, copied);
+			if (rc)
+				return rc;
+			if (msg->msg_flags & MSG_DONTWAIT)
+				return copied ? copied : -EAGAIN;
+			if ((smc->sk.sk_err) ||
+			    (time_after_eq(jiffies, max_wait_for_tx_space)))
+				return copied ? copied : -EPIPE;
+			if (smc_check_peer_gone(smc->conn.lgr)) {
+				smc->sk.sk_err = EPIPE;
+				release_sock(&smc->sk);
+				smc_terminate_conn(smc->conn.lgr);
+				lock_sock(&smc->sk);
+				return copied ? copied : -EPIPE;
+			}
+			goto again;
+		}
+	}
+	if (smc->sk.sk_err ||
+	    smc->sk.sk_shutdown & SEND_SHUTDOWN)
+		return -EPIPE;
+
+	writespace = atomic_read(&conn->tx_buf_space);
+	rc = smc_give_up_send(smc, copied);
+	if (rc)
+		return rc;
+
+	if (msg->msg_flags & MSG_OOB) {
+		if (writespace < len) {
+			conn->local_tx_ctrl.p_flags.urg_data_pending = 1;
+			rc = smc_ctrl_send(conn, conn->lnk_idx);
+			conn->local_tx_ctrl.p_flags.urg_data_pending = 0;
+		}
+	}
+
+	copy = min_t(size_t, to_send, writespace);
+	/* determine start of tx_buf */
+	prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	tx_cnt_prep = prep.s.curs.c;
+	buf = conn->rmb_tx_elem->rmb_tx;
+	/* determine tx_buf pieces - bottom and top of tx_buf */
+	if (tx_cnt_prep + copy <= conn->rmb_tx_size) {
+		tx_top = 0;
+		tx_bot = copy;
+		if (memcpy_from_msg(buf + tx_cnt_prep, msg, copy))
+			return -EFAULT;
+	} else {
+		tx_bot = conn->rmb_tx_size - tx_cnt_prep;
+		tx_top = copy - tx_bot;
+		if (memcpy_from_msg(buf + tx_cnt_prep, msg, tx_bot))
+			return -EFAULT;
+		if (memcpy_from_msg(buf, msg, tx_top))
+			return -EFAULT;
+	}
+	smc_curs_add(conn->rmb_tx_size, &prep.s.curs, copy);
+	atomic64_set(&conn->tx_curs_prep.s.acurs, prep.s.lcurs);
+	smp_mb__before_atomic();
+	atomic_sub(copy, &conn->tx_buf_space);
+	smp_mb__after_atomic();
+
+	if (msg->msg_flags & MSG_OOB)
+		atomic64_set(&conn->tx_urg_curs.s.acurs, prep.s.lcurs);
+
+	if ((tp->nonagle & TCP_NAGLE_CORK || msg->msg_flags & MSG_MORE) &&
+	    (writespace > copy)) {
+		if (!timer_pending(&smc->cork_timer))
+			/* cancelled at the beginning ? */
+			mod_timer(&smc->cork_timer,
+				  jiffies + msecs_to_jiffies(200));
+	} else {
+		if (atomic_read(&conn->rx_buf_space))
+			smc_write_data(smc);
+		else if (!delayed_work_pending(&smc->write_work))
+			schedule_delayed_work(&smc->write_work, 0);
+	}
+
+	copied = copied + copy;
+	to_send = to_send - copy;
+	if (copied < len)
+		goto again;
+
+	return copied;
+}
+
+static inline void smc_fill_sge(int *num, struct tx_sge sge[], int offset1,
+				int len1, int offset2, int len2)
+{
+	memset(sge, 0, 2 * sizeof(struct tx_sge));
+	sge[0].offset = offset1;
+	sge[0].len = len1;
+	if (len2) {
+		*num = 2;
+		sge[1].offset = offset2;
+		sge[1].len = len2;
+	} else {
+		*num = 1;
+	}
+}
+
+static inline void smc_write_data_adapt(struct smc_connection *conn,
+					struct smc_curs *prod,
+					struct smc_curs *sent, int len)
+{
+	smc_curs_add(conn->peer_rx_buf_len, &prod->s.curs, len);
+	smp_mb__before_atomic();
+	atomic_sub(len, &conn->rx_buf_space);
+	smp_mb__after_atomic();
+	smc_curs_add(conn->rmb_tx_size, &sent->s.curs, len);
+}
+
+void smc_write_data(struct smc_sock *smc)
+{
+	struct smc_link *lnk;
+	struct smc_connection *conn = &smc->conn;
+	int copy, space_left1, space_left2, send_len, to_send;
+	int writespace;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs sent;
+	struct smc_curs prep;
+	struct tx_sge sge[2];
+	int tx_top = 0, tx_bot = 0;
+	int tx_top1 = 0, tx_top2 = 0;
+	int tx_bot1 = 0, tx_bot2 = 0;
+	int rc = 0, num, urgent = 0;
+	u8 lnk_idx;
+
+again:
+	lnk_idx = conn->lnk_idx;
+	if (conn->lgr->lnk_down[lnk_idx]) {
+		release_sock(&smc->sk);
+		msleep(1000);
+		lock_sock(&smc->sk);
+		goto again;
+	}
+	lnk = &conn->lgr->lnk[lnk_idx];
+	sent.s.lcurs = atomic64_read(&conn->tx_curs_sent.s.acurs);
+	prep.s.lcurs = atomic64_read(&conn->tx_urg_curs.s.acurs);
+	if (!prep.s.curs.w && !prep.s.curs.c)
+		prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	else
+		urgent = 1;
+	to_send = smc_curs_diff(conn->rmb_tx_size, &sent, &prep);
+	if (!to_send)
+		goto send_ctrl;
+	writespace = atomic_read(&conn->rx_buf_space);
+	if (writespace <= 0)
+		return;
+	if (find_first_zero_bit(lnk->pending_bit_mask, lnk->send_wr_num) ==
+		lnk->send_wr_num) {
+		schedule_delayed_work(&smc->write_work, (HZ / 10));
+		return;
+	}
+	copy = min_t(size_t, to_send, writespace);
+	if (sent.s.curs.c + copy <= conn->rmb_tx_size) {
+		tx_top = 0;
+		tx_bot = copy;
+	} else {
+		tx_bot = conn->rmb_tx_size - sent.s.curs.c;
+		tx_top = copy - tx_bot;
+	}
+	prod.s.lcurs = atomic64_read(&conn->local_tx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left1 = conn->peer_rx_buf_len - prod.s.curs.c;
+		space_left2 = cons.s.curs.c;
+
+		send_len = min_t(size_t, copy, space_left1);
+		if (send_len <= tx_bot) {
+			tx_bot1 = send_len;
+			tx_bot2 = tx_bot - tx_bot1;
+			tx_top1 = 0;
+			tx_top2 = tx_top;
+		} else {
+			tx_bot1 = tx_bot;
+			tx_bot2 = 0;
+			tx_top1 = send_len - tx_bot;
+			tx_top2 = tx_top - tx_top1;
+		}
+		smc_fill_sge(&num, sge, sent.s.curs.c, tx_bot1, 0, tx_top1);
+		rc = smc_rdma_write(conn, prod.s.curs.c, lnk_idx, num, sge);
+		if (rc)
+			return;
+		copy -= send_len;
+		smc_write_data_adapt(conn, &prod, &sent, send_len);
+
+		if (copy && space_left2 && (tx_bot2 + tx_top2 > 0)) {
+			send_len = min_t(size_t, copy, space_left2);
+			if (tx_bot2 > send_len) {
+				tx_bot2 = send_len;
+				tx_top2 = 0;
+			} else {
+				if (tx_bot2 + tx_top2 > send_len)
+					tx_top2 = send_len - tx_bot2;
+			}
+			if (tx_bot2)
+				smc_fill_sge(&num, sge, sent.s.curs.c,
+					     tx_bot2, tx_top1, tx_top2);
+			else if (tx_top2)
+				smc_fill_sge(&num, sge, tx_top1, tx_top2, 0, 0);
+			rc = smc_rdma_write(conn, 0, lnk_idx, num, sge);
+			if (rc)
+				return;
+			smc_write_data_adapt(conn, &prod, &sent,
+					     tx_bot2 + tx_top2);
+		}
+	} else {
+		space_left1 = cons.s.curs.c - prod.s.curs.c;
+		send_len = min_t(size_t, copy, space_left1);
+		if (send_len <= tx_bot) {
+			tx_bot = send_len;
+			tx_top = 0;
+		} else {
+			if ((send_len - tx_bot) <= tx_top)
+				tx_top = send_len - tx_bot;
+		}
+		smc_fill_sge(&num, sge, sent.s.curs.c, tx_bot, 0, tx_top);
+		rc = smc_rdma_write(conn, prod.s.curs.c, lnk_idx, num, sge);
+		if (rc)
+			return;
+		smc_write_data_adapt(conn, &prod, &sent, send_len);
+	}
+	atomic64_set(&conn->local_tx_ctrl.p_curs.s.acurs, prod.s.lcurs);
+	atomic64_set(&conn->tx_curs_sent.s.acurs, sent.s.lcurs);
+
+	/* send cursor updates and signal peer */
+	if (to_send > atomic_read(&conn->rx_buf_space))
+		conn->local_tx_ctrl.p_flags.write_blocked = 1;
+	else
+		conn->local_tx_ctrl.p_flags.write_blocked = 0;
+
+	if (urgent) {
+		conn->local_tx_ctrl.p_flags.urg_data_pending = 1;
+		conn->local_tx_ctrl.p_flags.urg_data_present = 1;
+	}
+
+send_ctrl:
+	rc = smc_ctrl_send(conn, lnk_idx);
+	if (rc)
+		/* failover is scheduled */
+		return;
+
+	if (urgent) {
+		conn->local_tx_ctrl.p_flags.urg_data_pending = 0;
+		conn->local_tx_ctrl.p_flags.urg_data_present = 0;
+		atomic64_set(&conn->tx_urg_curs.s.acurs, 0);
+			/* could be lost in case of failover */
+	}
+
+	prep.s.lcurs = atomic64_read(&conn->tx_urg_curs.s.acurs);
+	if (!prep.s.curs.w && !prep.s.curs.c)
+		prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	if (smc_curs_diff(conn->rmb_tx_size, &sent, &prep) &&
+	    (smc->sk.sk_state != SMC_CLOSED)) {
+		schedule_delayed_work(&smc->write_work, 0);
+	} else {
+		if (wq_has_sleeper(smc->sk.sk_wq) &&
+		    atomic_read(&conn->tx_buf_space) &&
+		    test_bit(SOCK_NOSPACE, &smc->sk.sk_socket->flags))
+			smc_sock_wake_tx(&smc->sk);
+		if (smc->sk.sk_state != SMC_ACTIVE)
+			wake_up(&smc->prep_waiter);
+	}
+}
+
+void smc_write_worker(struct work_struct *work)
+{
+	int rc;
+
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+		write_work.work);
+
+	lock_sock(&smc->sk);
+again:
+	if (smc_prepared_sends(&smc->conn) == 0)
+		goto out;
+	rc = smc_wait_rx_space(smc);
+	if (rc < 0)
+		goto out;
+	if (!rc) {
+		if (!smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close &&
+		    !smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close &&
+		    (smc->sk.sk_state != SMC_CLOSED))
+			goto again;
+		else
+			goto out;
+	}
+	if (!smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close &&
+	    !smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close &&
+	    (smc->sk.sk_state != SMC_CLOSED))
+		smc_write_data(smc);
+out:
+	release_sock(&smc->sk);
+}
+
+void smc_send_close_alt(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) &&
+		    (i != smc->conn.lnk_idx))
+			break;
+	}
+	if (i <= SMC_MAX_SYM_LINKS)
+		smc_ctrl_send(&smc->conn, i);
+}
+
+int smc_send_close(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int rc;
+
+	if (smc_to_read(smc) > 0)
+		smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	else
+		smc->conn.local_tx_ctrl.conn_state_flags.closed_conn = 1;
+	if (lgr->lgr_type != NONE) {
+		rc = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		if (rc && (lgr->lgr_type != SINGLE)) {
+			/* try other link */
+			smc_send_close_alt(smc);
+		}
+	}
+	wake_up_interruptible(&smc->rx_waiter);
+	return 0;
+}
+
+int smc_to_read(struct smc_sock *smc)
+{
+	int rc;
+	struct smc_curs prod; /* to receive */
+	struct smc_curs cons; /* already consumed */
+	struct smc_curs spl_cons; /* spliced data in flight */
+	struct smc_curs urg;
+
+	if ((smc->conn.rx_urg_state != SMC_URG_VALID) &&
+	    !smc_splice_in_use(smc))
+		return atomic_read(&smc->conn.bytes_to_rcv);
+
+	/* local rx cursor consumer_c is up to date */
+	prod.s.lcurs = atomic64_read(&smc->conn.local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+
+	if ((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&smc->conn.rx_urg_curs.s.acurs);
+		if (smc_curs_diff(smc->conn.rmb_rx_size, &cons, &urg))
+			prod.s.lcurs =
+				atomic64_read(&smc->conn.rx_urg_curs.s.acurs);
+	}
+	if (smc_splice_in_use(smc)) {
+		spl_cons.s.lcurs =
+			atomic64_read(&smc->conn.local_rx_spliced.s.acurs);
+		if (spl_cons.s.curs.reserved0)
+			cons = spl_cons;
+	}
+
+	rc = smc_curs_diff(smc->conn.rmb_rx_size, &cons, &prod);
+	return (rc == -EALREADY) ? 0 : rc;
+}
+
+static void smc_update_cons_curs(struct smc_sock *smc, struct smc_link *lnk)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0, to_confirm;
+	struct smc_curs cfed;
+	struct smc_curs cons;
+
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	cfed.s.lcurs = atomic64_read(&conn->rx_curs_confirmed.s.acurs);
+	to_confirm = smc_curs_diff(conn->rmb_rx_size, &cfed, &cons);
+
+	if (conn->local_rx_ctrl.p_flags.cons_curs_upd_req ||
+	    ((to_confirm > conn->upd_cons_limit) &&
+	     ((to_confirm / (conn->rmb_rx_size / 2) > 0) ||
+	      conn->local_rx_ctrl.p_flags.write_blocked ||
+	      conn->local_rx_ctrl.p_flags.urg_data_pending))) {
+		if (lnk)
+			rc = smc_ctrl_send(conn, lnk - &conn->lgr->lnk[0]);
+		else
+			rc = smc_ctrl_send(conn, conn->lnk_idx);
+		if (rc)
+			return;
+		atomic64_set(&conn->rx_curs_confirmed.s.acurs,
+			    atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs));
+		conn->local_rx_ctrl.p_flags.cons_curs_upd_req = 0;
+	}
+	if (conn->local_rx_ctrl.p_flags.write_blocked &&
+	    (!smc_to_read(smc) > 0))
+		conn->local_rx_ctrl.p_flags.write_blocked = 0;
+}
+
+static void smc_pipe_buf_noop(struct pipe_inode_info *pipe,
+			      struct pipe_buffer *buf)
+{
+	;
+}
+
+static int smc_pipe_buf_steal(struct pipe_inode_info *pipe,
+			      struct pipe_buffer *buf)
+{
+	return 1;
+}
+
+struct smc_pipe_buf_priv {
+	struct smc_sock *smc;
+	unsigned int     len;
+};
+
+static void smc_pipe_buf_release(struct pipe_inode_info *pipe,
+				 struct pipe_buffer *buf)
+{
+	struct smc_curs curs;
+	struct smc_pipe_buf_priv *priv
+		= (struct smc_pipe_buf_priv *)buf->private;
+	struct smc_sock *smc = priv->smc;
+
+	/* Update cursors when we reached the last page */
+	curs.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+	smc_curs_add(smc->conn.rmb_rx_size, &curs.s.curs, priv->len);
+	atomic64_set(&smc->conn.local_tx_ctrl.c_curs.s.acurs, curs.s.lcurs);
+	smc_update_cons_curs(smc, NULL);
+	wake_up(&smc->splice_waiter);
+	kfree((void *)buf->private);
+}
+
+static const struct pipe_buf_operations smc_pipe_ops = {
+	.can_merge = 0,
+	.confirm = generic_pipe_buf_confirm,
+	.get = smc_pipe_buf_noop,
+	.release = smc_pipe_buf_release,
+	.steal = smc_pipe_buf_steal
+};
+
+static void smc_splice_spd_release(struct splice_pipe_desc *spd, unsigned int i)
+{
+	;
+}
+
+static void smc_free_spd(struct smc_sock *smc)
+{
+	if (smc->spd) {
+		kfree(smc->spd->partial);
+		smc->spd->partial = NULL;
+		kfree(smc->spd->pages);
+		smc->spd->pages = NULL;
+		kfree(smc->spd);
+		smc->spd = NULL;
+	}
+}
+
+static int smc_alloc_spd(struct smc_sock *smc)
+{
+	int msize = smc->conn.rmb_rx_size / PAGE_SIZE + 1;
+
+	smc->spd = kzalloc(sizeof(*smc->spd), GFP_KERNEL);
+	if (!smc->spd)
+		return -ENOMEM;
+	smc->spd->pages = kcalloc(msize, sizeof(struct page), GFP_KERNEL);
+	if (!smc->spd->pages)
+		goto error;
+	smc->spd->partial = kcalloc(msize, sizeof(struct partial_page),
+				    GFP_KERNEL);
+	if (!smc->spd->partial)
+		goto error;
+	smc->spd->nr_pages_max = msize;
+	smc->spd->ops = &smc_pipe_ops;
+	smc->spd->spd_release = smc_splice_spd_release;
+
+	return 0;
+
+error:
+	smc_free_spd(smc);
+	return -ENOMEM;
+}
+
+/* no extra handling of splice options required:
+ *   SPLICE_F_MOVE doesn't apply and can be ignored
+ *   SPLICE_F_NONBLOCK is handled by splice_to_pipe
+ *   SPLICE_F_MORE doesn't apply (fdout is a pipe)
+ *   SPLICE_F_GIFT is unused
+ */
+static int smc_splice_to_pipe(struct smc_sock *smc,
+			      struct pipe_inode_info *pipe,
+			      unsigned int flags, void *src, int len)
+{
+	struct partial_page *ppage;
+	struct splice_pipe_desc *spd;
+	struct smc_pipe_buf_priv *priv;
+	int rc, i;
+
+	if (!len)
+		return 0;
+	if (!smc->spd && smc_alloc_spd(smc))
+		return -ENOMEM;
+	spd = smc->spd;
+	spd->nr_pages = 0;
+	spd->flags = flags;
+
+	while (len) {
+		spd->pages[spd->nr_pages] = virt_to_page(src);
+		ppage = &spd->partial[spd->nr_pages];
+		ppage->offset = (unsigned long)src & ~PAGE_MASK;
+		ppage->len = min_t(int, PAGE_SIZE - ppage->offset, len);
+		src += ppage->len;
+		len -= ppage->len;
+		priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+		if (!priv)
+			goto error;
+		priv->smc = smc;
+		priv->len = ppage->len;
+		ppage->private = (unsigned long)priv;
+		spd->nr_pages++;
+	}
+
+	rc = splice_to_pipe(pipe, spd);
+	if (rc < 0 && smc->sk.sk_err)
+		goto error;
+	return rc;
+
+error:
+	for (i = 0; i < spd->nr_pages; ++i)
+		kfree((void *)spd->partial[i].private);
+	return -EFAULT;
+}
+
+/* Wait for data to arrive
+ *   @smc    smc socket
+ *   @len    num bytes to wait for
+ *   @wall   don't return until full request is fulfilled
+ *   @timeo  max seconds to wait, 0 for no timeout
+ * Returns boolean value or -EAGAIN in case timeout expired.
+ */
+int smc_wait_rx_data(struct smc_sock *smc, int len, bool wall,
+		     long timeo)
+{
+	int rc;
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int target;
+	DEFINE_WAIT(wait);
+
+	if ((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID)
+		target = 1;
+	else
+		target = sock_rcvlowat(sk, wall, len);
+	if (smc_to_read(smc) >= target)
+		return 1;
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+	set_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
+	rc = sk_wait_event(sk, &timeo,
+			   (sk->sk_err != 0) ||
+			   sk->sk_shutdown & RCV_SHUTDOWN ||
+			   sock_flag(sk, SOCK_DONE) ||
+			   (smc_to_read(smc) >= target) ||
+			   smc_stop_received(conn));
+	clear_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
+	finish_wait(sk_sleep(sk), &wait);
+	return ((rc || timeo) ? rc : -EAGAIN);
+}
+
+static inline void smc_skip_urg_data(struct smc_sock *smc,
+				     struct smc_curs *cons,
+				     struct smc_curs *urg)
+{
+	if (((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) &&
+	    (cons->s.curs.c == urg->s.curs.c) &&
+	    (cons->s.curs.w == urg->s.curs.w))
+		smc->conn.rx_urg_state |= SMC_URG_RECV;
+}
+
+/* derived from tcp_recv_urg */
+static int smc_conn_recv_urg(struct smc_sock *smc, struct msghdr *msg,
+			     size_t len, unsigned int flags)
+{
+	int rc = 0;
+
+	if (sock_flag(&smc->sk, SOCK_URGINLINE) ||
+	    !(smc->conn.rx_urg_state & SMC_URG_VALID) ||
+	    (smc->conn.rx_urg_state & SMC_URG_READ))
+		return -EINVAL;
+
+	if (!(flags & MSG_PEEK)) {
+		smc->conn.rx_urg_state |= SMC_URG_READ;
+		atomic_dec(&smc->conn.bytes_to_rcv);
+	}
+
+	msg->msg_flags |= MSG_OOB;
+
+	if (len > 0) {
+		if (!(flags & MSG_TRUNC))
+			rc = memcpy_to_msg(msg, &smc->conn.rx_urg_data, 1);
+		len = 1;
+	} else {
+		msg->msg_flags |= MSG_TRUNC;
+	}
+	if (!sock_flag(&smc->sk, SOCK_URGINLINE)) {
+		struct smc_curs cons;
+
+		/* may cause smc_curs_diff errors, which are acceptable here */
+		cons.s.lcurs =
+			atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+		smc_curs_add(smc->conn.rmb_rx_size, &cons.s.curs, 1);
+		atomic64_set(&smc->conn.local_tx_ctrl.c_curs.s.acurs,
+			     cons.s.lcurs);
+	}
+
+	return rc ? -EFAULT : len;
+}
+
+int smc_conn_recv(struct smc_sock *smc, struct msghdr *msg, size_t len,
+		  int flags)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+	int toread = 0, space_left, copy, buffer_len;
+	int target;		/* Read at least these many bytes */
+	char *buffer;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs urg;
+	struct smc_curs spl_cons;
+	long timeo;
+	int loop = 0;
+
+	msg->msg_namelen = 0;
+
+	if (smc->sk.sk_state == SMC_INIT)
+		return -ENOTCONN;
+
+	buffer_len = conn->rmb_rx_size;
+	buffer =  conn->rmb_rx_elem->rmb_rx;
+	len = min_t(size_t, len, conn->rmb_rx_size);
+
+	if (flags & MSG_OOB)
+		return smc_conn_recv_urg(smc, msg, len, flags);
+
+again:
+	target = sock_rcvlowat(&smc->sk, flags & MSG_WAITALL, len);
+	timeo = sock_rcvtimeo(&smc->sk, flags & MSG_DONTWAIT);
+	if (signal_pending(current)) {
+		rc = timeo ? -EINTR : -EAGAIN;
+		return rc;
+	}
+	rc = smc_wait_rx_data(smc, target, flags & MSG_WAITALL, timeo);
+	if ((rc == -EAGAIN) || (rc == -EINTR))
+		return rc;
+	if (!rc) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			release_sock(&smc->sk);
+			smc_terminate_conn(smc->conn.lgr);
+			lock_sock(&smc->sk);
+			return -EINTR;
+		}
+		goto again;
+	}
+	if (smc_splice_in_use(smc)) {
+		if (smc_splice_in_progress(smc) && (flags &  MSG_DONTWAIT))
+			return -EAGAIN;
+		wait_event_interruptible(smc->splice_waiter,
+					 !smc_splice_in_progress(smc));
+	}
+	toread = smc_to_read(smc);
+	rc = 0;
+
+	if ((toread <= 0) &&
+	    (smc->sk.sk_err ||
+	     smc->sk.sk_shutdown & RCV_SHUTDOWN ||
+	     sock_flag(&smc->sk, SOCK_DONE) ||
+	     smc_stop_received(conn)))
+		return sock_error(&smc->sk);
+
+	prod.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	if (toread <= 0)
+		goto check_repeat;
+
+	if ((conn->rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+		if (smc_curs_diff(conn->rmb_rx_size, &cons, &urg))
+			prod = urg;
+		else /* data preceding urg already received */
+			conn->rx_urg_state |= SMC_URG_RECV;
+	} else {
+		urg.s.lcurs = 0;
+	}
+
+	if ((toread < target) && !smc_stop_received(conn))
+		goto check_repeat;
+
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left = prod.s.curs.c - cons.s.curs.c;
+		copy = min_t(size_t, len, space_left);
+		if (!(flags & MSG_TRUNC)) {
+			if (memcpy_to_msg(msg, buffer + cons.s.curs.c, copy))
+				return -EFAULT;
+		}
+		len -= copy;
+		rc += copy;
+	} else {
+		space_left = buffer_len - cons.s.curs.c;
+		if (space_left) {
+			copy = min_t(size_t, len, space_left);
+			if (!(flags & MSG_TRUNC)) {
+				if (memcpy_to_msg(msg, buffer + cons.s.curs.c,
+						  copy))
+					return -EFAULT;
+			}
+			len -= copy;
+			rc += copy;
+		}
+		if (len) {
+			space_left = prod.s.curs.c;
+			copy = min_t(size_t, len, space_left);
+			if (!(flags & MSG_TRUNC)) {
+				if (memcpy_to_msg(msg, buffer, copy))
+					return -EFAULT;
+			}
+			len -= copy;
+			rc += copy;
+		}
+	}
+
+	/* update cursors */
+	if (!(flags &  MSG_PEEK)) {
+		smc_curs_add(buffer_len, &cons.s.curs, rc);
+		smc_skip_urg_data(smc, &cons, &urg);
+		smp_mb__before_atomic();
+		atomic_sub(rc, &conn->bytes_to_rcv);
+		smp_mb__after_atomic();
+		atomic64_set(&conn->local_tx_ctrl.c_curs.s.acurs, cons.s.lcurs);
+		spl_cons.s.lcurs =
+			atomic64_read(&conn->local_rx_spliced.s.acurs);
+		if (spl_cons.s.curs.reserved0)
+			atomic64_set(&conn->local_rx_spliced.s.acurs,
+				     cons.s.lcurs);
+		/* send consumer cursor update if required */
+		smc_update_cons_curs(smc, NULL);
+	}
+check_repeat:
+	if ((toread < target) &&
+	    !smc_stop_received(conn) &&
+	    !conn->local_tx_ctrl.conn_state_flags.abnormal_close) {
+		loop = 0;
+		goto again;
+	}
+
+	return rc;
+}
+
+ssize_t smc_conn_splice_read(struct smc_sock *smc, struct pipe_inode_info *pipe,
+			     size_t len, unsigned int flags)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0, ret = 0;
+	int toread = 0, space_left, copy, buffer_len;
+	int target = 1;		/* read at least these many bytes */
+	char *buffer;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs spl_cons;
+	struct smc_curs urg;
+	long timeo;
+	int loop = 0;
+
+	if (smc->sk.sk_state == SMC_INIT)
+		return -ENOTCONN;
+
+	buffer_len = conn->rmb_rx_size;
+	buffer =  conn->rmb_rx_elem->rmb_rx;
+	len = min_t(size_t, len, conn->rmb_rx_size);
+
+again:
+	timeo = sock_rcvtimeo(&smc->sk, 0);
+	if (signal_pending(current)) {
+		rc = timeo ? -EINTR : -EAGAIN;
+		return rc;
+	}
+	rc = smc_wait_rx_data(smc, target, 0, timeo);
+	if ((rc == -EAGAIN) || (rc == -EINTR))
+		return rc;
+	if (!rc) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			release_sock(&smc->sk);
+			smc_terminate_conn(smc->conn.lgr);
+			lock_sock(&smc->sk);
+			return -EINTR;
+		}
+		goto again;
+	}
+	toread = smc_to_read(smc);
+
+	if ((toread <= 0) &&
+	    (smc->sk.sk_err ||
+	     smc->sk.sk_shutdown & RCV_SHUTDOWN ||
+	     sock_flag(&smc->sk, SOCK_DONE) ||
+	     smc_stop_received(conn)))
+		return sock_error(&smc->sk);
+
+	rc = 0;
+	prod.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	if (toread <= 0)
+		goto check_repeat;
+
+	if ((conn->rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+		if (smc_curs_diff(conn->rmb_rx_size, &cons, &urg))
+			prod = urg;
+		else /* data preceding urg already received */
+			conn->rx_urg_state |= SMC_URG_RECV;
+	} else {
+		urg.s.lcurs = 0;
+	}
+
+	if ((toread < target) && !smc_stop_received(conn))
+		goto check_repeat;
+
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left = prod.s.curs.c - cons.s.curs.c;
+		copy = min_t(size_t, len, space_left);
+		rc = smc_splice_to_pipe(smc, pipe, flags,
+					buffer + cons.s.curs.c, copy);
+		if (rc < 0)
+			return -EFAULT;
+		len -= rc;
+	} else {
+		space_left = buffer_len - cons.s.curs.c;
+		if (space_left) {
+			copy = min_t(size_t, len, space_left);
+			rc = smc_splice_to_pipe(smc, pipe, flags,
+						buffer + cons.s.curs.c,
+						copy);
+			if (rc < 0)
+				return -EFAULT;
+			len -= rc;
+		}
+		if (len) {
+			space_left = prod.s.curs.c;
+			copy = min_t(size_t, len, space_left);
+			ret = smc_splice_to_pipe(smc, pipe, flags, buffer,
+						 copy);
+			if (ret < 0)
+				return -EFAULT;
+			len -= ret;
+			rc += ret;
+		}
+	}
+
+	/* update cursors */
+	spl_cons.s.lcurs = atomic64_read(&conn->local_rx_spliced.s.acurs);
+	if (!spl_cons.s.curs.reserved0) {
+		/* no previous splice in progress, initialize to current */
+		spl_cons = cons;
+		spl_cons.s.curs.reserved0 = 1;
+	}
+	smc_curs_add(buffer_len, &spl_cons.s.curs, rc);
+	smc_skip_urg_data(smc, &spl_cons, &urg);
+	atomic64_set(&conn->local_rx_spliced.s.acurs, spl_cons.s.lcurs);
+	/* we update cursors in a callback after splicing completed */
+check_repeat:
+	if ((toread < target) &&
+	    !smc_stop_received(conn) &&
+	    !conn->local_tx_ctrl.conn_state_flags.abnormal_close) {
+		loop = 0;
+		goto again;
+	}
+
+	return rc;
+}
+
+void smc_check_qp_attr(struct smc_link *lnk)
+{
+	int rc;
+	struct ib_qp_attr *attr = &lnk->qp_attr;
+	struct ib_qp_init_attr init_attr;
+
+	memset(attr, 0, sizeof(struct ib_qp_attr));
+	memset(&init_attr, 0, sizeof(struct ib_qp_init_attr));
+	rc = ib_query_qp(lnk->roce_qp, attr,
+			 IB_QP_STATE |
+			 IB_QP_CUR_STATE |
+			 IB_QP_PKEY_INDEX |
+			 IB_QP_PORT |
+			 IB_QP_QKEY |
+			 IB_QP_AV |
+			 IB_QP_PATH_MTU |
+			 IB_QP_TIMEOUT |
+			 IB_QP_RETRY_CNT |
+			 IB_QP_RNR_RETRY |
+			 IB_QP_RQ_PSN |
+			 IB_QP_ALT_PATH |
+			 IB_QP_MIN_RNR_TIMER |
+			 IB_QP_SQ_PSN |
+			 IB_QP_PATH_MIG_STATE |
+			 IB_QP_CAP |
+			 IB_QP_VID |
+			 IB_QP_DEST_QPN,
+			 &init_attr);
+
+	lnk->send_wr_num = min_t(size_t, smc_ctrl_buf_cnt,
+				 lnk->qp_attr.cap.max_send_wr);
+	lnk->recv_wr_num = min_t(size_t, smc_ctrl_buf_cnt * 10,
+				 lnk->qp_attr.cap.max_recv_wr);
+}
+
+void smc_check_port_attr(struct smc_ib_device *smc_ibdev, int port_num)
+{
+	struct ib_device *dev = smc_ibdev->dev;
+	struct ib_port_attr *pattr;
+	int rc;
+
+	pattr = &smc_ibdev->pattr[port_num];
+	memset(pattr, 0, sizeof(struct ib_port_attr));
+	rc = ib_query_port(dev, port_num + 1, pattr);
+}
+
+void smc_check_dev_attr(struct smc_ib_device *smc_ibdev)
+{
+	struct ib_device *dev = smc_ibdev->dev;
+	struct ib_device_attr *attr = &smc_ibdev->attr;
+	int i, rc;
+
+	memset(attr, 0, sizeof(struct ib_device_attr));
+	rc = ib_query_device(dev, attr);
+
+	for (i = 0; i < 2; i++)
+		smc_check_port_attr(smc_ibdev, i);
+}
+
diff --git a/net/smc/smc_llc.c b/net/smc/smc_llc.c
new file mode 100644
index 0000000..7f6eb2a
--- /dev/null
+++ b/net/smc/smc_llc.c
@@ -0,0 +1,1597 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  SMC-R Link Layer Control
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Klaus Wacker <Klaus.Wacker@de.ibm.com>
+ *              Ursula Braun <ursula.braun@de.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/wait.h>
+#include <linux/kthread.h>
+#include <linux/hardirq.h>
+#include <linux/types.h>
+#include <net/tcp.h>
+#include <net/sock.h>
+#ifdef CONFIG_S390
+#include <asm/debug.h>
+#endif
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <linux/splice.h>
+#include <linux/mm.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+static enum llc_msg_group llc_msg_to_grp(enum llc_msg_type type)
+{
+	enum llc_msg_group grp;
+
+	switch (type) {
+	case LLC_ADD_LINK:
+		grp = LLC_GRP_ADD_LINK;
+		break;
+	case LLC_DEL_LINK:
+		grp = LLC_GRP_DEL_LINK;
+		break;
+	case LLC_CONFIRM_RKEY:
+		grp = LLC_GRP_CONF_RKEY;
+		break;
+	case LLC_TEST_LINK:
+		grp = LLC_GRP_TEST_LINK;
+		break;
+	default:
+		grp = LLC_GRP_NONE;
+	}
+	return grp;
+}
+
+int llc_enqueue(struct smc_link *lnk, struct llc_msg *msg)
+{
+	struct smc_link_group *lgr;
+	struct llc_qentry *qentry;
+	unsigned long flags;
+
+	lgr = lnk->lnk_grp;
+	qentry = kzalloc(sizeof(*qentry), GFP_ATOMIC);
+	if (!qentry)
+		return -ENOMEM;
+	qentry->link = lnk;
+	memcpy(&qentry->msg, msg, sizeof(struct llc_msg));
+	if (msg->hd.flags & LLC_FLAG_RESP) {
+		switch (msg->hd.type) {
+		case LLC_ADD_LINK:
+		case LLC_DEL_LINK:
+		case LLC_CONFIRM_LINK:
+		case LLC_ADD_LINK_CONT:
+			lgr->llc_ctl[LLC_SERV].qentry = qentry;
+			break;
+		case LLC_CONFIRM_RKEY:
+		case LLC_CONFIRM_RKEY_CONT:
+		case LLC_DELETE_RKEY:
+			lgr->llc_ctl[lgr->role].qentry = qentry;
+			break;
+		case LLC_TEST_LINK:
+			lgr->llc_ctl[LLC_TESTLINK].qentry = qentry;
+			break;
+		default:
+			llc_send_del_link(lnk, NULL, 0, LLC_DEL_PROT_VIOL,
+					  LLC_NO_NOTIFY);
+			smc_terminate_conn(lgr);
+			kfree(qentry);
+			break;
+		}
+		wake_up_interruptible(&lgr->llc_waiter);
+		return 0;
+	}
+	spin_lock_irqsave(&lgr->llc_event_q_lock, flags);
+	list_add_tail(&qentry->list, &lgr->llc_event_q);
+	spin_unlock_irqrestore(&lgr->llc_event_q_lock, flags);
+	if (lgr->llc_wq)
+		queue_work(lgr->llc_wq, &lgr->llc_event_work);
+
+	return 0;
+}
+
+int llc_initiate(struct smc_link_group *lgr, enum llc_msg_group llc_grp)
+{
+	enum llc_msg_group llc_grp_allowed;
+	int peer;
+	enum llc_ctl_ind llc_seq;
+	int rest_time = 1;
+
+	llc_grp_allowed = (llc_grp == LLC_GRP_CONF_RKEY) ?
+				LLC_GRP_CONF_RKEY : LLC_GRP_NONE;
+	peer = !lgr->role;
+	switch (llc_grp) {
+	case LLC_GRP_CONF_RKEY:
+		llc_seq = (enum llc_ctl_ind)lgr->role;
+		break;
+	case LLC_GRP_TEST_LINK:
+		llc_seq = 2;
+		peer = 2;
+		llc_grp_allowed = LLC_GRP_TEST_LINK;
+		break;
+	default:
+		llc_seq = LLC_SERV;
+	}
+reserve_llc:
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	if ((lgr->llc_ctl[llc_seq].active == LLC_GRP_NONE) &&
+	    (lgr->llc_ctl[peer].active <= llc_grp_allowed)) {
+		lgr->llc_ctl[llc_seq].active = llc_grp;
+		spin_unlock_bh(&lgr->llc_ctl_lock);
+		return rest_time;
+	}
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	rest_time = wait_event_interruptible_timeout(lgr->llc_waiter,
+		((lgr->llc_ctl[llc_seq].active == LLC_GRP_NONE) &&
+		 (lgr->llc_ctl[peer].active <= llc_grp_allowed)),
+		LLC_WAIT_TIMEO);
+	if (!rest_time)
+		return rest_time;
+	goto reserve_llc;
+}
+
+void llc_stop(struct smc_link_group *lgr, int llc_grp)
+{
+	enum llc_ctl_ind llc_seq;
+
+	switch (llc_grp) {
+	case LLC_GRP_CONF_RKEY:
+		llc_seq = (enum llc_ctl_ind)lgr->role;
+		break;
+	case LLC_GRP_TEST_LINK:
+		llc_seq = 2;
+		break;
+	default:
+		llc_seq = LLC_SERV;
+	}
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	memset(&lgr->llc_ctl[llc_seq], 0, sizeof(struct llc_ctl));
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	if (lgr->delayed_q &&
+	    (llc_grp != LLC_GRP_TEST_LINK)) {
+		queue_work(lgr->llc_wq, &lgr->llc_event_work);
+	} else {
+		wake_up_interruptible(&lgr->llc_waiter);
+	}
+}
+
+struct llc_qentry *llc_wait(struct smc_link_group *lgr, enum smc_role initiator,
+			    int time_out, u8 exp_msg)
+{
+	struct llc_ctl *ctl = &lgr->llc_ctl[initiator];
+
+	wait_event_interruptible_timeout(lgr->llc_waiter,
+					 (ctl->qentry ||
+					 ((ctl->ctrl_elem == LLC_SEND_POSTED) &&
+					  ctl->wc_status) ||
+					 (lgr->lgr_type == NONE)),
+					 time_out);
+	if (IS_ERR(ctl->qentry) ||	/* waiter has been killed */
+	    !ctl->qentry) {
+		ctl->qentry = NULL;
+		goto out;
+	}
+	if (((initiator == SMC_SERV) ||
+	     (exp_msg != LLC_CONFIRM_LINK)) &&
+	    (ctl->ctrl_elem != LLC_SEND_POSTED)) {
+		/* cancel post req when send completes after llc msg arrival */
+		lgr->lnk[ctl->lnk_idx].pending[ctl->ctrl_elem].post_ctl =
+			LLC_NO_NOTIFY; /* mark as cancelled */
+	}
+	if ((exp_msg) && (ctl->qentry->msg.hd.type != exp_msg)) {
+		if ((exp_msg == LLC_ADD_LINK) &&
+		    (ctl->qentry->msg.hd.type == LLC_DEL_LINK)) {
+			schedule_work(&lgr->llc_del_link_work);
+			return NULL;
+		}
+		kfree(ctl->qentry);
+		ctl->qentry = NULL;
+	}
+out:
+	return ctl->qentry;
+}
+
+static int llc_add_pending_send(struct smc_link *link,
+				enum llc_ctl_ind ctl_idx, u32 *elem)
+{
+	u64 wr_id;
+	int rc = 0;
+
+	wr_id = (u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	rc = smc_get_ctrl_buf(link, elem);
+	if (rc < 0)
+		return rc;
+	memset(&link->pending[*elem], 0, sizeof(struct pending_send));
+	link->pending[*elem].wr_id_send = wr_id;
+	link->pending[*elem].post_ctl = ctl_idx;
+	if (ctl_idx != LLC_NO_NOTIFY)
+		link->lnk_grp->llc_ctl[ctl_idx].ctrl_elem = *elem;
+	link->send_wr[*elem].wr_id = wr_id;
+	return 0;
+}
+
+static inline int llc_send_msg(struct smc_link *link, struct llc_msg *msg,
+			       enum llc_ctl_ind ctl_idx)
+{
+	struct smc_link_group *lgr;
+	u32 index = link->send_wr_num;
+	int rc;
+
+	if ((atomic_read(&link->state) != SMC_LINK_UP) || !link->lnk_grp)
+		return -EPIPE;
+	lgr = link->lnk_grp;
+	if (lgr->lgr_type == NONE)
+		return -EPIPE;
+	if (ctl_idx != LLC_NO_NOTIFY)
+		lgr->llc_ctl[ctl_idx].lnk_idx = link - &lgr->lnk[0];
+	/* send llc message */
+	rc = llc_add_pending_send(link, ctl_idx, &index);
+	if (!rc)
+		rc = smc_wr_send(link, NULL, (u64)msg, index);
+	if (rc) {
+		if (index < link->send_wr_num)
+			test_and_clear_bit(index, link->pending_bit_mask);
+	}
+	return rc;
+}
+
+int llc_send_confirm_link(struct smc_link *link, u8 mac[], union ib_gid *gid,
+			  enum llc_reqresp reqresp, enum llc_ctl_ind ctl_idx)
+{
+	struct llc_confirm_msg confllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	u32 index;
+	int rc = 0;
+
+	memset(&confllc, 0, sizeof(confllc));
+	confllc.hd.type = LLC_CONFIRM_LINK;
+	confllc.hd.length = sizeof(struct llc_confirm_msg);
+	if (reqresp == LLC_RESP)
+		confllc.hd.flags = 0x80;
+	else
+		confllc.hd.flags = 0x00;
+	confllc.hd.flags |= SMC_CACHE_ALIGN_SIZE_SHORT;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	memcpy(confllc.sender_mac, mac, 6);
+	memcpy(confllc.sender_gid, gid, 16);
+	confllc.max_links = lgr->max_links;
+	confllc.link_num = link->link_id;
+	confllc.link_uid = htonl(lgr->lgr_id + link->link_id);
+
+	hton_three(link->roce_qp->qp_num, confllc.sender_qp_num);
+	/* send llc message */
+	if (ctl_idx != LLC_NO_NOTIFY)
+		lgr->llc_ctl[ctl_idx].lnk_idx = link - &lgr->lnk[0];
+	rc = llc_add_pending_send(link, ctl_idx, &index);
+	if (!rc)
+		rc = smc_wr_send(link, NULL, (u64)&confllc, index);
+	return rc;
+}
+
+int llc_send_add_link(struct smc_link *link, struct smc_roce_defs *rocdefs,
+		      struct smc_link *linkw, enum llc_reqresp reqresp,
+		      enum llc_ctl_ind ctl_idx)
+{
+	struct llc_add_link_msg addllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int rc = 0;
+
+	memset(&addllc, 0, sizeof(addllc));
+	addllc.hd.type = LLC_ADD_LINK;
+	addllc.hd.length = sizeof(struct llc_add_link_msg);
+	if (reqresp == LLC_RESP) {
+		addllc.hd.flags = 0x80;
+		addllc.flags2 = min(linkw->path_mtu, linkw->mtu_peer);
+	} else {
+		addllc.hd.flags = 0x00;
+		if (linkw)
+			addllc.flags2 = linkw->path_mtu;
+		else
+			addllc.flags2 = link->path_mtu;
+		if (lgr->role == SMC_SERV)
+			lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	}
+	memcpy(addllc.sender_mac, &rocdefs->mac, 6);
+	memcpy(addllc.sender_gid, &rocdefs->gid, 16);
+	if (linkw) {
+		addllc.link_num = linkw->link_id;
+		hton_three(linkw->roce_qp->qp_num, addllc.sender_qp_num);
+		hton_three(linkw->psn_initial, addllc.initial_psn);
+	}
+	/* send llc message */
+	rc = llc_send_msg(link, (struct llc_msg *)&addllc, ctl_idx);
+	return rc;
+}
+
+int llc_send_del_link(struct smc_link *link, struct smc_link *link2,
+		      u8 ord_flag, u32 reason, enum llc_ctl_ind ctl_idx)
+{
+	struct llc_del_link_msg delllc;
+
+	memset(&delllc, 0, sizeof(delllc));
+	delllc.hd.type = LLC_DEL_LINK;
+	delllc.hd.length = sizeof(struct llc_del_link_msg);
+	delllc.hd.flags = ord_flag;
+	if (link2) {
+		delllc.link_num = link2->link_id;
+	} else {
+		delllc.link_num = 0;
+		delllc.hd.flags |= LLC_FLAG_DEL_ALL;
+	}
+	delllc.reason = htonl(reason);
+
+	return llc_send_msg(link, (struct llc_msg *)&delllc, ctl_idx);
+}
+
+int llc_do_confirm_rkey(struct smc_link *link, struct rmb_rx_addrs *rmb_rx_elem)
+{
+	struct llc_confirm_rkey_msg confrkeyllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int prim_link_id, rt_idx, i, prim_lnk_idx;
+
+	memset(&confrkeyllc, 0, sizeof(confrkeyllc));
+	confrkeyllc.hd.type = LLC_CONFIRM_RKEY;
+	confrkeyllc.hd.length = sizeof(struct llc_confirm_rkey_msg);
+	confrkeyllc.hd.flags = 0x00;
+	lgr->llc_ctl[lgr->role].qentry = NULL;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	confrkeyllc.rt[0].rmb_key =
+		htonl(rmb_rx_elem->mr_rx[prim_lnk_idx]->rkey);
+	confrkeyllc.rt[0].rmb_vaddr =
+		cpu_to_be64((u64)rmb_rx_elem->rmb_rx_dma[prim_lnk_idx]);
+
+	prim_link_id = link->link_id;
+	rt_idx = 1;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((lgr->lnk[i].link_id == prim_link_id) ||
+		    (atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP))
+			continue;
+		confrkeyllc.rt[rt_idx].link_id = lgr->lnk[i].link_id;
+		confrkeyllc.rt[rt_idx].rmb_key =
+			htonl(rmb_rx_elem->mr_rx[i]->rkey);
+		confrkeyllc.rt[rt_idx].rmb_vaddr =
+			cpu_to_be64((u64)rmb_rx_elem->rmb_rx_dma[i]);
+		rt_idx = rt_idx + 1;
+	}
+	confrkeyllc.rt[0].link_id = rt_idx - 1;
+	if (rt_idx == 1) /* nothing to confirm */
+		WARN_ON(1);
+
+	return llc_send_msg(link, (struct llc_msg *)&confrkeyllc,
+			    (enum llc_ctl_ind)lgr->role);
+}
+
+int llc_do_del_rkey(struct smc_link *link, struct rmb_rx_addrs *rmb_rx_elem)
+{
+	struct llc_delete_rkey_msg delrkeyllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int prim_lnk_idx;
+
+	memset(&delrkeyllc, 0, sizeof(delrkeyllc));
+	delrkeyllc.hd.type = LLC_DELETE_RKEY;
+	delrkeyllc.hd.length = sizeof(struct llc_delete_rkey_msg);
+	delrkeyllc.hd.flags = 0x00;
+	lgr->llc_ctl[lgr->role].qentry = NULL;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	delrkeyllc.num_rkeys = 1;
+	delrkeyllc.rk[0] =
+		htonl(rmb_rx_elem->mr_rx[prim_lnk_idx]->rkey);
+
+	return llc_send_msg(link, (struct llc_msg *)&delrkeyllc,
+			    (enum llc_ctl_ind)lgr->role);
+}
+
+struct bufs_list_pos {
+	struct list_head	*head;
+	int			ind;
+	struct rmb_rx_addrs	*elem;
+};
+
+static void llc_get_next_rx_rmb_slot(struct smc_link_group *lgr,
+				     struct bufs_list_pos *bufs_pos)
+{
+	int start;
+
+	start = bufs_pos->ind;
+	if (bufs_pos->elem && bufs_pos->elem->list.next !=
+		&bufs_pos->head[start]) {
+		bufs_pos->elem =
+			(struct rmb_rx_addrs *)bufs_pos->elem->list.next;
+		return;
+	}
+	while (start < (SMC_RMBE_SIZE_VALUES - 1)) {
+		start++;
+		if (bufs_pos->head[start].next !=
+			&bufs_pos->head[start]) {
+			bufs_pos->elem = (struct rmb_rx_addrs *)
+						bufs_pos->head[start].next;
+			bufs_pos->ind = start;
+			return;
+		}
+	}
+}
+
+static int llc_add_link_cont(struct smc_link *link, int lnk_idx, u8 link_num,
+			     u8 *num_rkeys_todo, struct bufs_list_pos *pos)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_add_link_cont_msg addc_llcs;
+	int prim_lnk_idx, i;
+	u8 n;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	addc_llcs.link_num = link_num;
+	addc_llcs.num_rkeys = *num_rkeys_todo;
+	n = *num_rkeys_todo;
+	memset(&addc_llcs.reserved2, 0,
+	       sizeof(struct llc_add_link_cont_msg) -
+		      offsetof(struct llc_add_link_cont_msg, reserved2));
+	for (i = 0; i < min_t(u8, n, 2); i++) {
+next_rmb_slot:
+		llc_get_next_rx_rmb_slot(lgr, pos);
+		if (!pos->elem) {
+			addc_llcs.num_rkeys = addc_llcs.num_rkeys -
+							*num_rkeys_todo;
+			*num_rkeys_todo = 0;
+			break;
+		}
+		if (!pos->elem->mr_rx[prim_lnk_idx])
+			goto next_rmb_slot;
+		addc_llcs.rt[i].rmb_key =
+			htonl(pos->elem->mr_rx[prim_lnk_idx]->rkey);
+		addc_llcs.rt[i].rmb_key_new =
+			htonl(pos->elem->mr_rx[lnk_idx]->rkey);
+		addc_llcs.rt[i].rmb_vaddr_new =
+			cpu_to_be64((u64)pos->elem->rmb_rx_dma[lnk_idx]);
+		(*num_rkeys_todo)--;
+	}
+	addc_llcs.hd.type = LLC_ADD_LINK_CONT;
+	addc_llcs.hd.length = sizeof(struct llc_msg);
+	addc_llcs.hd.version = SMC_CLC_V1;
+	if (lgr->role == SMC_SERV)
+		addc_llcs.hd.flags = 0x00;
+	else
+		addc_llcs.hd.flags = 0x80;
+	llc_send_msg(link, (struct llc_msg *)&addc_llcs, LLC_SERV);
+	return 0;
+}
+
+int llc_send_test_link(struct smc_link *link, u8 udata[])
+{
+	struct llc_test_link_msg testllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+
+	testllc.hd.type = LLC_TEST_LINK;
+	memset(&testllc.user_data, 0, sizeof(testllc.user_data));
+	strncpy(testllc.user_data, udata, sizeof(testllc.user_data));
+	testllc.hd.length = sizeof(struct llc_test_link_msg);
+	testllc.hd.flags = 0x00;
+	lgr->llc_ctl[LLC_TESTLINK].qentry = NULL;
+	testllc.hd.version = SMC_CLC_V1;
+	return llc_send_msg(link, (struct llc_msg *)&testllc, LLC_TESTLINK);
+}
+
+int llc_get_fail_cause(struct smc_link_group *lgr, enum llc_ctl_ind initiator,
+		       struct smc_sock *smc)
+{
+	struct smc_decline_clc_msg dclc;
+	int reason_code, rc;
+
+	if ((initiator < 3) &&
+	    ((lgr->llc_ctl[initiator].ctrl_elem != LLC_SEND_POSTED) ||
+	     !lgr->llc_ctl[initiator].wc_status)) {
+		/* llc send failure */
+		return -ECOMM;
+	}
+	if (!smc)
+		return -ETIMEDOUT;
+	/* clc decl received or llc timeout */
+	memset(&dclc, 0, sizeof(dclc));
+	rc = clc_wait_msg(smc, SMC_CLC_NONBLOCK, (char *)&dclc,
+			  sizeof(struct smc_decline_clc_msg),
+			  SMC_CLC_DECLINE, &reason_code);
+	/* clc decline received */
+	return rc;
+}
+
+static u8 llc_count_rx_rmb_slot(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct rmb_rx_addrs *entry_rx;
+	int i;
+	u8 count;
+
+	count = 0;
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(entry_rx, &lgr->rmb_rx_bufs[i], list) {
+			if (entry_rx->mr_rx[lnk_idx])
+				count++;
+		}
+	}
+	return count;
+}
+
+int smc_find_rtoken_by_link(struct smc_link_group *lgr, int lnk_idx, u32 rkey)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		if ((lgr->rtok[i].rkey[lnk_idx] == rkey) &&
+		    (lgr->rtok[i].link_id[lnk_idx] ==
+						lgr->lnk[lnk_idx].link_id) &&
+		    (atomic_read(&lgr->rtok[i].active) == 1))
+			return i;
+	}
+	return -ENOENT;
+}
+
+static void llc_set_rtoken(struct smc_link_group *lgr, int link_n1, int link_n2,
+			   struct rmb_rtoken2 *rtoken)
+{
+	u32 rkey;
+	int rtok_idx;
+
+	rkey = ntohl(rtoken->rmb_key);
+	rtok_idx = smc_find_rtoken_by_link(lgr, link_n1, rkey);
+	if (rtok_idx == -ENOENT)
+		return;
+	lgr->rtok[rtok_idx].rkey[link_n2] = ntohl(rtoken->rmb_key_new);
+	lgr->rtok[rtok_idx].vaddr[link_n2] = be64_to_cpu(rtoken->rmb_vaddr_new);
+	lgr->rtok[rtok_idx].link_id[link_n2] = lgr->lnk[link_n2].link_id;
+}
+
+int smc_get_rtoken(struct smc_link_group *lgr)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		if (atomic_cmpxchg(&lgr->rtok[i].active, 0, 1) == 0)
+			return i;
+	}
+	return -ENOSPC;
+}
+
+static void llc_set_rtoken_by_link(struct smc_link_group *lgr, int link_n1,
+				   int rtoken_idx, struct rmb_rtoken3 *rtoken)
+{
+	int lnk_idx, found = 0;
+
+	for (lnk_idx = 0; lnk_idx <= SMC_MAX_SYM_LINKS; lnk_idx++) {
+		if (lgr->lnk[lnk_idx].link_id == link_n1) {
+			found = 1;
+			break;
+		}
+	}
+	if (!found) {
+		WARN_ON(1);
+		return;
+	}
+	lgr->rtok[rtoken_idx].rkey[lnk_idx] = ntohl(rtoken->rmb_key);
+	lgr->rtok[rtoken_idx].vaddr[lnk_idx] = be64_to_cpu(rtoken->rmb_vaddr);
+	lgr->rtok[rtoken_idx].link_id[lnk_idx] = link_n1;
+}
+
+static int smc_clear_rtokens_by_link(struct smc_link_group *lgr, int link_n1,
+				     __be32 rmb_rkey)
+{
+	int i, lnk_idx, rtoken_idx, found = 0;
+	u32 rkey;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id == link_n1) {
+			found = 1;
+			break;
+		}
+	}
+	if (!found)
+		return -ENODEV;
+	lnk_idx = i;
+	rkey = ntohl(rmb_rkey);
+	rtoken_idx = smc_find_rtoken_by_link(lgr, lnk_idx, rkey);
+	found = (rtoken_idx < 0 ? 0 : 1);
+	if (!found)
+		return -ENODEV;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		lgr->rtok[rtoken_idx].rkey[i] = 0;
+		lgr->rtok[rtoken_idx].vaddr[i] = 0;
+		lgr->rtok[rtoken_idx].link_id[i] = 0;
+	}
+	atomic_set(&lgr->rtok[rtoken_idx].active, 0);
+	return 0;
+}
+
+static int llc_alloc_alt_link(struct smc_link_group *lgr,
+			      enum lgr_type lgr_new_t)
+{
+	int i, lnk_idx, total_links = 0;
+
+	if ((lgr->lgr_type != SINGLE) &&
+	    ((lgr_new_t == ASYMMETRIC_LOCAL) ||
+	     (lgr_new_t == ASYMMETRIC_PEER)))
+		return -EMLINK;
+	if ((lgr_new_t == ASYMMETRIC_LOCAL) ||
+	    (lgr_new_t == ASYMMETRIC_PEER)) {
+		lnk_idx = SMC_MAX_SYM_LINKS;
+		for (i = SMC_MAX_SYM_LINKS; i > 0; i--) {
+			if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_FREED) {
+				lnk_idx = i;
+				break;
+			}
+		}
+	} else {
+		lnk_idx = SMC_MAX_SYM_LINKS + 1;
+		total_links = 0;
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_FREED) {
+				if (lnk_idx > SMC_MAX_SYM_LINKS)
+					lnk_idx = i;
+			} else {
+				total_links++;
+			}
+		}
+		if ((lgr->lgr_type == SYMMETRIC) &&
+		    (total_links >= SMC_MAX_SYM_LINKS))
+			return -EMLINK;
+		if (lnk_idx > SMC_MAX_SYM_LINKS)
+			return -EMLINK;
+	}
+	return lnk_idx;
+}
+
+static void llc_deny_req(struct llc_qentry *qentry, u8 flag, u8 retc,
+			 enum llc_ctl_ind ctl_idx)
+{
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	qentry->msg.hd.flags |= flag;
+	qentry->msg.hd.version |= retc;
+	llc_send_msg(qentry->link, &qentry->msg, ctl_idx);
+	kfree(qentry);
+}
+
+static void llc_save_add_link_info(struct smc_link *link,
+				   struct llc_add_link_msg *add_llc)
+{
+	memcpy(link->gid_peer, ((union ib_gid *)add_llc->sender_gid)->raw, 16);
+	memcpy(link->mac_peer, add_llc->sender_mac, 6);
+	ntoh_three(&link->qp_num_peer, add_llc->sender_qp_num);
+	ntoh_three(&link->psn_peer, add_llc->initial_psn);
+	link->mtu_peer = add_llc->flags2 & 0x0f;
+}
+
+static int llc_cli_rkey_exchange(struct smc_link *link,
+				 struct smc_link *link_new,
+				 struct smc_sock *smc, int lnk_idx)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_cont_msg *addc_llc;
+	struct bufs_list_pos pos = {
+		.head = lgr->rmb_rx_bufs,
+		.ind = -1,
+		.elem = NULL,
+	};
+	int prim_lnk_idx, i;
+	u8 n, num_rkeys_send, num_rkeys_recv;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	num_rkeys_send = llc_count_rx_rmb_slot(lgr, prim_lnk_idx);
+	do {
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK_CONT);
+		if (!qentry) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return llc_get_fail_cause(lgr, LLC_SERV, smc);
+		}
+		addc_llc = (struct llc_add_link_cont_msg *)&qentry->msg;
+		num_rkeys_recv = addc_llc->num_rkeys;
+		n = addc_llc->num_rkeys;
+		for (i = 0; i < min_t(u8, n, 2); i++) {
+			llc_set_rtoken(lgr, prim_lnk_idx, lnk_idx,
+				       &addc_llc->rt[i]);
+			num_rkeys_recv--;
+		}
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		kfree(qentry);
+		llc_add_link_cont(link, lnk_idx, link_new->link_id,
+				  &num_rkeys_send, &pos);
+	} while (num_rkeys_send || num_rkeys_recv);
+	return 0;
+}
+
+static int llc_cli_conf_link(struct smc_link *link,
+			     struct smc_roce_defs *rocedef,
+			     struct smc_link *link_new, int lnk_idx,
+			     enum lgr_type lgr_new_t)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_del_link_msg *del_llc;
+	int rc = 0, cas;
+
+	/* receive CONFIRM LINK request over RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO, 0);
+	if (!qentry) {
+		rc = llc_send_del_link(link, link_new, 0,
+				       LLC_DEL_LOST_PATH,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	if (qentry->msg.hd.type == LLC_CONFIRM_LINK) {
+		rc = smc_modify_qp_rts(link_new);
+		cas = 0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+		if (rc || !(qentry->msg.hd.flags & 0xf) ||
+		    !lgr->cache_align_size || (lgr->cache_align_size != cas)) {
+			/* no cache alignment specified, decline */
+			kfree(qentry);
+			rc = llc_send_del_link(link, link_new, 0,
+					       LLC_DEL_PROT_VIOL,
+					       LLC_NO_NOTIFY);
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return -ENOLINK;
+		}
+		if (!lgr->cache_align_size)
+			lgr->cache_align_size = cas;
+		smc_check_qp_attr(link_new);
+		/* send CONFIRM LINK response over RoCE fabric */
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		rc = llc_send_confirm_link(link_new,
+					   (u8 *)&rocedef->mac,
+					   &rocedef->gid, LLC_RESP,
+					   LLC_NO_NOTIFY);
+		kfree(qentry);
+		if (rc) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			rc = llc_send_del_link(link, link_new, 0,
+					       LLC_DEL_LOST_PATH,
+					       LLC_NO_NOTIFY);
+			if (rc && smc_lnk_downing(&link->state))
+				smc_link_down(lgr, link - &lgr->lnk[0]);
+			return -ENOLINK;
+		}
+		atomic_set(&link_new->state, SMC_LINK_UP);
+		lgr->lgr_type = lgr_new_t;
+	} else {
+		del_llc = (struct llc_del_link_msg *)&qentry->msg;
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		qentry->msg.hd.flags |= LLC_FLAG_RESP;
+		llc_send_msg(link, &qentry->msg, LLC_SERV);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		kfree(qentry);
+		return -ENOLINK;
+	}
+	return 0;
+}
+
+int llc_cli_add_link(struct smc_link *link, struct llc_qentry *qentry,
+		     struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_link *link_new;
+	struct llc_add_link_msg *add_llc;
+	struct smc_roce_defs rocedef;
+	enum lgr_type lgr_new_t = SYMMETRIC;
+	int lnk_idx, rc;
+	int reuse_lnk_idx = SMC_MAX_SYM_LINKS + 1;
+
+	rocedef.ibdev = NULL;
+	smc_find_alt_roce_resources(&rocedef, lgr, link);
+	add_llc = (struct llc_add_link_msg *)&qentry->msg;
+	if (!memcmp(add_llc->sender_gid, link->gid_peer, 16) &&
+	    !memcmp(add_llc->sender_mac, link->mac_peer, 6)) {
+		if (!rocedef.ibdev) {
+			llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+				     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+			return 0;
+		}
+		lgr_new_t = ASYMMETRIC_PEER;
+	}
+	if (rocedef.ibdev) {
+		if (!smc_port_active(rocedef.ibdev, rocedef.port)) {
+			llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+				     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+			return 0;
+		}
+	} else {
+		lgr_new_t = ASYMMETRIC_LOCAL;
+		memcpy(&rocedef, &link->roce_defs,
+		       sizeof(struct smc_roce_defs));
+		reuse_lnk_idx = link - &lgr->lnk[0];
+	}
+	lnk_idx = llc_alloc_alt_link(lgr, lgr_new_t);
+	if (lnk_idx < 0) {
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		return 0;
+	}
+	rc = smc_create_link(lgr, lnk_idx, &rocedef);
+	if (rc) {
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		return 0;
+	}
+	link_new = &lgr->lnk[lnk_idx];
+	llc_save_add_link_info(link_new, add_llc);
+	rc = smc_ready_link(link_new);
+	if (rc) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		smc_free_link(lgr, lnk_idx);
+		return 0;
+	}
+	link_new->link_id = add_llc->link_num;
+	kfree(qentry);
+	smc_map_rmbs_to_link(lgr, lnk_idx, reuse_lnk_idx);
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	llc_send_add_link(link, &rocedef, link_new, LLC_RESP, LLC_SERV);
+	rc = llc_cli_rkey_exchange(link, link_new, smc, lnk_idx);
+	if (!rc)
+		rc = llc_cli_conf_link(link, &rocedef, link_new, lnk_idx,
+				       lgr_new_t);
+	return rc;
+}
+
+static int llc_srv_rkey_exchange(struct smc_link *link,
+				 struct smc_link *link_new,
+				 struct smc_sock *smc,
+				 int lnk_idx)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_cont_msg *addc_llc;
+	struct bufs_list_pos pos = {
+		.head = lgr->rmb_rx_bufs,
+		.ind = -1,
+		.elem = NULL,
+	};
+	int prim_lnk_idx;
+	int i;
+	u8 n, num_rkeys_send, num_rkeys_recv;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	num_rkeys_send = llc_count_rx_rmb_slot(lgr, prim_lnk_idx);
+	do {
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		llc_add_link_cont(link, lnk_idx, link_new->link_id,
+				  &num_rkeys_send, &pos);
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK_CONT);
+		if (!qentry) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return llc_get_fail_cause(lgr, LLC_SERV, smc);
+		}
+		addc_llc = (struct llc_add_link_cont_msg *)&qentry->msg;
+		num_rkeys_recv = addc_llc->num_rkeys;
+		n = addc_llc->num_rkeys;
+		for (i = 0; i < min_t(u8, n, 2); i++) {
+			llc_set_rtoken(lgr, prim_lnk_idx, lnk_idx,
+				       &addc_llc->rt[i]);
+			num_rkeys_recv--;
+		}
+		kfree(qentry);
+	} while (num_rkeys_send || num_rkeys_recv);
+	return 0;
+}
+
+static int llc_srv_conf_link(struct smc_link *link,
+			     struct smc_roce_defs *rocedef,
+			     struct smc_link *link_new, int lnk_idx,
+			     enum lgr_type lgr_new_t)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	int cas, rc;
+
+	/* send CONFIRM LINK request over the RoCE fabric */
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	rc = llc_send_confirm_link(link_new, (u8 *)&rocedef->mac,
+				   &rocedef->gid, LLC_REQ, LLC_SERV);
+	/* receive CONFIRM LINK response over the RoCE fabric */
+	if (!rc)
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_CONFIRM_LINK);
+	if (rc || !qentry) {
+		/* send DELETE LINK */
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		rc = llc_send_del_link(link, link_new, 0, LLC_DEL_LOST_PATH,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		kfree(qentry);
+		if (rc) {
+			smc_terminate_conn(lgr);
+			return rc;
+		}
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	cas = 0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+	if (!(qentry->msg.hd.flags & 0xf) ||
+	    !lgr->cache_align_size || (lgr->cache_align_size != cas)) {
+		/* no cache alignment specified, decline */
+		kfree(qentry);
+		rc = llc_send_del_link(link, link_new, 0,
+				       LLC_DEL_PROT_VIOL,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		if (rc) {
+			smc_terminate_conn(lgr);
+			return rc;
+		}
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	if (!lgr->cache_align_size)
+		lgr->cache_align_size = cas;
+	atomic_set(&link_new->state, SMC_LINK_UP);
+	lgr->lgr_type = lgr_new_t;
+	kfree(qentry);
+	return 0;
+}
+
+int llc_srv_add_link(struct smc_link *link, struct smc_roce_defs *rocedef,
+		     struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_link *link_new;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_msg *add_llc;
+	enum lgr_type lgr_new_t = SYMMETRIC;
+	int lnk_idx, rc = 0;
+	int reuse_lnk_idx = SMC_MAX_SYM_LINKS + 1;
+
+	if (rocedef->ibdev) {
+		if (!smc_port_active(rocedef->ibdev, rocedef->port))
+			return -ENOLINK;
+	} else {
+		lgr_new_t = ASYMMETRIC_LOCAL;
+		memcpy(rocedef, &link->roce_defs, sizeof(struct smc_roce_defs));
+		reuse_lnk_idx = link - &lgr->lnk[0];
+	}
+	lnk_idx = llc_alloc_alt_link(lgr, lgr_new_t);
+	if (lnk_idx < 0)
+		return 0;
+	rc = smc_create_link(lgr, lnk_idx, rocedef);
+	if (rc)
+		return rc;
+	link_new = &lgr->lnk[lnk_idx];
+	rc = llc_send_add_link(link, rocedef, link_new, LLC_REQ, LLC_SERV);
+	if (!rc) {
+		/* receive ADD LINK response over the RoCE fabric */
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK);
+	}
+	if (rc || !qentry) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		return llc_get_fail_cause(lgr, LLC_SERV, smc);
+	}
+	add_llc = (struct llc_add_link_msg *)&qentry->msg;
+	if (add_llc->hd.flags & LLC_FLAG_ADD_LNK_REJ) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		kfree(qentry);
+		return -ENOLINK;
+	}
+	if ((lgr->lgr_type == SINGLE) &&
+	    (!memcmp(add_llc->sender_gid, link->gid_peer, 16) &&
+	    !memcmp(add_llc->sender_mac, link->mac_peer, 6)))
+		lgr_new_t = ASYMMETRIC_PEER;
+	llc_save_add_link_info(link_new, add_llc);
+	kfree(qentry);
+	smc_ready_link(link_new);
+	smc_map_rmbs_to_link(lgr, lnk_idx, reuse_lnk_idx);
+	llc_srv_rkey_exchange(link, link_new, smc, lnk_idx);
+	rc = llc_srv_conf_link(link, rocedef, link_new, lnk_idx, lgr_new_t);
+	return rc;
+}
+
+static bool start_llc_agent(enum smc_role initiator,
+			    struct llc_qentry *qentry)
+{
+	struct smc_link_group *lgr = qentry->link->lnk_grp;
+
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	if (lgr->llc_ctl[initiator].active) {
+		if (((qentry->msg.hd.type == LLC_ADD_LINK) ||
+		     (qentry->msg.hd.type == LLC_DEL_LINK)) &&
+		    !lgr->delayed_q) {
+			lgr->delayed_q = qentry;
+		} else {
+			kfree(qentry);
+		}
+		spin_unlock_bh(&lgr->llc_ctl_lock);
+		return 0;
+	}
+	lgr->llc_ctl[initiator].active = llc_msg_to_grp(qentry->msg.hd.type);
+	if (qentry == lgr->delayed_q)
+		lgr->delayed_q = NULL;
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	lgr->llc_ctl[initiator].qentry = qentry;
+	return 1;
+}
+
+void llc_cancel_wait(struct smc_link *lnk)
+{
+	struct smc_link_group *lgr = lnk->lnk_grp;
+	int i;
+	bool cancelled = false;
+
+	for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+		u8 ctl_idx = lnk->pending[i].post_ctl;
+
+		if (!lnk->pending[i].conn &&
+		    (ctl_idx != LLC_NO_NOTIFY) &&
+		    lgr->llc_ctl[ctl_idx].active &&
+		    !lgr->llc_ctl[ctl_idx].qentry) {
+			lgr->llc_ctl[ctl_idx].qentry = ERR_PTR(-EPIPE);
+			cancelled = true;
+		}
+	}
+	if (cancelled)
+		wake_up_interruptible(&lgr->llc_waiter);
+}
+
+void llc_kill_waiters(struct smc_link_group *lgr)
+{
+	int i;
+	bool cancelled = false;
+
+	for (i = 0; i < LLC_NO_NOTIFY; i++) {
+		if (lgr->llc_ctl[i].active && !lgr->llc_ctl[i].qentry) {
+			lgr->llc_ctl[i].qentry = ERR_PTR(-EPIPE);
+			cancelled = true;
+		}
+	}
+	if (cancelled)
+		wake_up_interruptible(&lgr->llc_waiter);
+}
+
+static void llc_process_cli_add_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct smc_link *lnk;
+
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	mutex_lock(&lgr->conf_mutex);
+	llc_cli_add_link(lnk, qentry, NULL);
+	mutex_unlock(&lgr->conf_mutex);
+}
+
+static void llc_del_asym_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc_resp;
+	struct smc_link *link = &lgr->lnk[0];
+	int asym_lnk = SMC_MAX_SYM_LINKS + 1;
+	int i, j, k, rc;
+
+	/* determine asymmetric link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		for (j = i + 1; j <= SMC_MAX_SYM_LINKS; j++) {
+			if (!memcmp(lgr->lnk[i].roce_defs.gid.raw,
+				    lgr->lnk[j].roce_defs.gid.raw, 16)) {
+				asym_lnk = j; /* asym_lnk is i or j */
+				break;
+			}
+		}
+		if (asym_lnk < SMC_MAX_SYM_LINKS + 1)
+			break;
+	}
+	if (asym_lnk == SMC_MAX_SYM_LINKS + 1)
+		return; /* no asymmetric link */
+	for (k = 0; k <= SMC_MAX_SYM_LINKS; k++) {
+		if ((k != i) &&
+		    !memcmp(lgr->lnk[i].gid_peer, lgr->lnk[k].gid_peer, 16)) {
+			asym_lnk = i;
+			break;
+		}
+		if ((k != j) &&
+		    !memcmp(lgr->lnk[j].gid_peer, lgr->lnk[k].gid_peer, 16))
+			break;
+	}
+	if (k == SMC_MAX_SYM_LINKS + 1)
+		return; /* no asymmetric link */
+	if (asym_lnk == 0)
+		link = &lgr->lnk[1];
+	lgr->llc_ctl[LLC_SERV].active = LLC_GRP_DEL_LINK;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (!smc_lnk_downing(&lgr->lnk[asym_lnk].state))
+		return;
+	rc = smc_switch_conns(lgr, asym_lnk, 0);
+	smc_wait_no_pending_sends_on_link(&lgr->lnk[asym_lnk]);
+	if (rc == -ENOENT) {
+		smc_free_link(lgr, asym_lnk);
+		goto out;
+	}
+	rc = llc_send_del_link(link, &lgr->lnk[asym_lnk],
+			       LLC_FLAG_DEL_ORDERLY, LLC_DEL_NO_ASYM_NEEDED,
+			       LLC_SERV);
+	if (rc) {
+		if (smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+		goto out;
+	}
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO, LLC_DEL_LINK);
+	if (!qentry) {
+		if (smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+	} else {
+		del_llc_resp = (struct llc_del_link_msg *)&qentry->msg;
+		if ((ntohl(del_llc_resp->reason) != LLC_DEL_NO_ASYM_NEEDED) &&
+		    smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+		atomic_set(&lgr->lnk[asym_lnk].state, SMC_LINK_DOWN);
+	}
+	kfree(qentry);
+	smc_free_link(lgr, asym_lnk);
+out:
+	return;
+}
+
+static void llc_process_srv_add_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_add_link_fake_msg *add_llc;
+	struct smc_link *lnk;
+	struct smc_roce_defs rocdefs;
+	int rc;
+	enum lgr_type old_lgr_type;
+
+	rocdefs.ibdev = NULL;
+	rocdefs.port = 0;
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	add_llc = (struct llc_add_link_fake_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (qentry->msg.hd.flags & LLC_FLAG_LOCAL) {
+		rocdefs.ibdev = add_llc->ibdev;
+		rocdefs.port = add_llc->port_num;
+		rocdefs.vlan = lgr->vlan;
+		rc = smc_gid_by_dev(&rocdefs);
+		if (rc) {
+			kfree(qentry);
+			wake_up_interruptible(&lgr->llc_waiter);
+			return;
+		}
+	} else {
+		smc_find_alt_roce_resources(&rocdefs, lgr, lnk);
+	}
+	kfree(qentry);
+	qentry = NULL;
+	old_lgr_type = lgr->lgr_type;
+	mutex_lock(&lgr->conf_mutex);
+	rc = llc_srv_add_link(lnk, &rocdefs, NULL);
+	if (!rc && (lgr->lgr_type == SYMMETRIC)) {
+		/* delete the superfluos asymmetric link now */
+		llc_del_asym_link(lgr);
+	}
+	mutex_unlock(&lgr->conf_mutex);
+}
+
+void llc_process_add_link(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_add_link_work);
+
+	if (list_empty(&lgr->list) || (lgr->lgr_type == NONE)) {
+		/* link group to be deleted */
+		kfree(lgr->llc_ctl[LLC_SERV].qentry);
+		wake_up_interruptible(&lgr->llc_waiter);
+		goto out;
+	}
+	if (lgr->role == SMC_CLNT)
+		llc_process_cli_add_link(lgr);
+	else
+		llc_process_srv_add_link(lgr);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+out:
+	return;
+}
+
+static void llc_process_cli_del_all(struct smc_link_group *lgr,
+				    struct smc_link *link,
+				    struct llc_qentry *qentry)
+{
+	struct llc_del_link_msg *del_llc;
+	int empty;
+
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	del_llc->reason = 0;
+	mutex_lock(&lgr->conf_mutex);
+	llc_send_msg(link, &qentry->msg, LLC_SERV); /* send response */
+	memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+	mutex_unlock(&lgr->conf_mutex);
+	wake_up_interruptible(&lgr->llc_waiter);
+	/* delete link group at all */
+	spin_lock_bh(&smc_lgr_list.lock);
+	if (!list_empty(&lgr->list)) {
+		list_move_tail(&lgr->list, &smc_lgr_freeing_list);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		mutex_lock(&lgr->conf_mutex);
+		read_lock_bh(&lgr->conns_lock);
+		empty = RB_EMPTY_ROOT(&lgr->conns_all);
+		read_unlock_bh(&lgr->conns_lock);
+		mutex_unlock(&lgr->conf_mutex);
+		if (!empty)
+			smc_terminate_conn(lgr);
+		llc_kill_waiters(lgr);
+		cancel_work_sync(&lgr->llc_event_work);
+		smc_wait_no_pending_sends_on_link(link);
+		schedule_delayed_work(&lgr->free_lgr_final_work,
+				      2 * TCP_TIMEWAIT_LEN);
+	} else {
+		spin_unlock_bh(&smc_lgr_list.lock);
+	}
+}
+
+static void llc_process_cli_del_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc;
+	struct smc_link *lnk;
+	int i, link_found;
+
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	del_llc->hd.flags |= LLC_FLAG_RESP;
+	if (del_llc->hd.flags & LLC_FLAG_DEL_ALL) {
+		llc_process_cli_del_all(lgr, lnk, qentry);
+		goto out;
+	}
+	mutex_lock(&lgr->conf_mutex);
+	/* delete single link */
+	link_found = 0;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id != del_llc->link_num)
+			continue;
+		link_found = 1;
+		if (lnk->link_id == del_llc->link_num)
+			lgr->lgr_type = NONE; /* delete link of received llc */
+		else
+			llc_send_msg(lnk, &qentry->msg, LLC_SERV); /* response*/
+		if (smc_lnk_downing(&lgr->lnk[i].state)) {
+			smc_switch_conns(lgr, i, 0);
+			smc_wait_no_pending_sends_on_link(&lgr->lnk[i]);
+		}
+		if (ntohl(del_llc->reason) != LLC_DEL_NO_ASYM_NEEDED) {
+			lgr->lgr_type = SINGLE;
+			del_llc->reason = 0;
+		}
+		memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+		smc_free_link(lgr, i);
+		mutex_unlock(&lgr->conf_mutex);
+		wake_up_interruptible(&lgr->llc_waiter);
+		break;
+	}
+	if (!link_found) {
+		del_llc->reason = htonl(SMC_LLC_DEL_NOLNK);
+		llc_send_msg(lnk, &qentry->msg, LLC_SERV);
+		memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+		mutex_unlock(&lgr->conf_mutex);
+		wake_up_interruptible(&lgr->llc_waiter);
+	}
+out:
+	kfree(qentry);
+}
+
+static void llc_process_srv_del_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc;
+	struct smc_link *lnk;
+	struct smc_roce_defs rocdefs;
+	int i, lnk_idx = -1, found = 0;
+
+	mutex_lock(&lgr->conf_mutex);
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (qentry->msg.hd.flags & LLC_FLAG_DEL_ALL) {
+		/* delete entire lgr */
+		goto out;
+	}
+	/* delete single link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id == del_llc->link_num) {
+			found = 1;
+			lnk_idx = i;
+		}
+	}
+	lgr->lgr_type = SINGLE;
+	if (found && smc_lnk_downing(&lgr->lnk[lnk_idx].state)) {
+		smc_switch_conns(lgr, lnk_idx, 0);
+		smc_wait_no_pending_sends_on_link(&lgr->lnk[lnk_idx]);
+	}
+	if ((lgr->lgr_type != NONE) &&
+	    (found || (qentry->msg.hd.type == 0x80 + LLC_DEL_LINK))) {
+		struct llc_qentry *qentry2;
+
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		qentry->msg.hd.type = LLC_DEL_LINK;
+		qentry->msg.hd.flags &= ~LLC_FLAG_LOCAL;
+		llc_send_msg(lnk, &qentry->msg, LLC_SERV);
+		qentry2 = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO,
+				   LLC_DEL_LINK);
+		kfree(qentry2);
+	}
+	if (found) {
+		smc_free_link(lgr, lnk_idx);
+		if (qentry && (lgr->lgr_type == SINGLE) &&
+		    !list_empty(&lgr->list)) {
+			/* setup asymm alt link */
+			rocdefs.ibdev = NULL;
+			smc_find_alt_roce_resources(&rocdefs, lgr, lnk);
+			if (!rocdefs.ibdev) {
+				lgr->llc_ctl[LLC_SERV].active =
+							LLC_GRP_ADD_LINK;
+				llc_srv_add_link(lnk, &rocdefs, NULL);
+			}
+		}
+	}
+out:
+	mutex_unlock(&lgr->conf_mutex);
+	kfree(qentry);
+	llc_stop(lgr, LLC_GRP_DEL_LINK);
+}
+
+void llc_process_del_link(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_del_link_work);
+
+	if (lgr->role == SMC_CLNT)
+		llc_process_cli_del_link(lgr);
+	else
+		llc_process_srv_del_link(lgr);
+}
+
+void llc_process_confirm_rkey(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+					    llc_conf_rkey_work);
+	struct smc_link *lnk;
+	struct llc_qentry *qentry;
+	struct llc_confirm_rkey_msg *rk_llc;
+	int rt_idx = 0, i, rc = 0;
+	enum llc_ctl_ind initiator;
+	u8 num_entries, more = 0;
+
+	initiator = (lgr->role == SMC_SERV) ? LLC_CLNT : LLC_SERV;
+	qentry = lgr->llc_ctl[initiator].qentry;
+	lnk = qentry->link;
+	rk_llc = (struct llc_confirm_rkey_msg *)&qentry->msg;
+	lgr->llc_ctl[initiator].qentry = NULL;
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	if (lgr->llc_ctl[LLC_SERV].active > LLC_GRP_CONF_RKEY) {
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_NEG;
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_RETRY;
+	} else {
+		num_entries = rk_llc->rt[0].link_id;
+		rt_idx = smc_get_rtoken(lgr);
+		llc_set_rtoken_by_link(lgr, lnk->link_id, rt_idx,
+				       &rk_llc->rt[0]);
+		for (i = 1; i <= min_t(u8, num_entries, 2); i++)
+			llc_set_rtoken_by_link(lgr, rk_llc->rt[i].link_id,
+					       rt_idx, &rk_llc->rt[i]);
+		more = (num_entries > 2) ? 1 : 0;
+	}
+	rc = llc_send_msg(lnk, &qentry->msg, LLC_NO_NOTIFY);
+	kfree(qentry);
+again:
+	if (more && (rc >= 0)) {
+		qentry = llc_wait(lgr, (enum smc_role)initiator, LLC_WAIT_TIMEO,
+				  LLC_CONFIRM_RKEY_CONT);
+		if (!qentry) {
+			more = 0;
+		} else {
+			struct llc_conf_rkey_cont_msg *rkc_llc =
+				(struct llc_conf_rkey_cont_msg *)&qentry->msg;
+			lgr->llc_ctl[initiator].qentry = NULL;
+			qentry->msg.hd.flags |= LLC_FLAG_RESP;
+			num_entries = rkc_llc->num_rkeys;
+			for (i = 0; i <= min_t(u8, num_entries, 2); i++) {
+				llc_set_rtoken_by_link(lgr,
+						       rkc_llc->rt[i].link_id,
+						       rt_idx, &rkc_llc->rt[i]);
+			}
+			more = (num_entries > 3) ? 1 : 0;
+			rc = llc_send_msg(lnk, &qentry->msg, LLC_NO_NOTIFY);
+			kfree(qentry);
+		}
+		goto again;
+	}
+	memset(&lgr->llc_ctl[initiator], 0, sizeof(struct llc_ctl));
+}
+
+void llc_process_delete_rkey(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_del_rkey_work);
+	struct smc_link *lnk;
+	struct llc_qentry *qentry;
+	struct llc_delete_rkey_msg *drk_llc;
+	int initiator;
+
+	initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+	qentry = lgr->llc_ctl[initiator].qentry;
+	lnk = qentry->link;
+	drk_llc = (struct llc_delete_rkey_msg *)&qentry->msg;
+	lgr->llc_ctl[initiator].qentry = NULL;
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	if (lgr->llc_ctl[LLC_SERV].active > LLC_GRP_CONF_RKEY) {
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_NEG;
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_RETRY;
+	} else {
+		int i, rc;
+		u8 local_mask = 0, err_val;
+
+		err_val = 0x80;
+		for (i = 0; i < min_t(u8, drk_llc->num_rkeys, 8); i++) {
+			rc = smc_clear_rtokens_by_link(lgr, lnk->link_id,
+						       drk_llc->rk[i]);
+			if (rc < 0)
+				local_mask = local_mask + err_val;
+			err_val = err_val / 2;
+		}
+		if (local_mask) {
+			drk_llc->hd.flags |= LLC_FLAG_RKEY_NEG;
+			drk_llc->err_mask = local_mask;
+		}
+	}
+	llc_send_msg(lnk, &qentry->msg, initiator);
+	kfree(qentry);
+	memset(&lgr->llc_ctl[initiator], 0, sizeof(struct llc_ctl));
+}
+
+static void llc_handle_event(struct llc_qentry *qentry,
+			     struct smc_link_group *lgr)
+{
+	struct smc_link *lnk = qentry->link;
+	struct llc_ctl *srv_ctl = &lgr->llc_ctl[LLC_SERV];
+	int initiator;
+
+	switch (qentry->msg.hd.type) {
+	case LLC_ADD_LINK:
+		if (list_empty(&lgr->list)) /* link group to be freed */
+			break;
+		if (lgr->role == SMC_CLNT) {
+			if (srv_ctl->active == LLC_GRP_ADD_LINK) {
+				/* some process is waiting for this response */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_add_link_work);
+			}
+		} else if (start_llc_agent(SMC_SERV, qentry)) {
+			/* else on SERV handle client suggest */
+			schedule_work(&lgr->llc_add_link_work);
+		}
+		break;
+	case LLC_DEL_LINK:
+		if (lgr->role == SMC_CLNT) {
+			if (srv_ctl->active) {
+				/* DEL LINK REQ during ADD LINK SEQ */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_del_link_work);
+			}
+		} else {
+			if (qentry->msg.hd.flags & LLC_FLAG_DEL_ALL) {
+				smc_terminate_conn(lgr);
+				kfree(qentry);
+			} else if ((srv_ctl->active == LLC_GRP_ADD_LINK) &&
+				   (!srv_ctl->qentry)) {
+				/* DEL LINK REQ during ADD LINK SEQ */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_del_link_work);
+			}
+		}
+		break;
+	case LLC_CONFIRM_RKEY:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		if (start_llc_agent(initiator, qentry))
+			schedule_work(&lgr->llc_conf_rkey_work);
+		break;
+	case LLC_DELETE_RKEY:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		if (start_llc_agent(initiator, qentry))
+			schedule_work(&lgr->llc_del_rkey_work);
+		break;
+	case LLC_CONFIRM_RKEY_CONT:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		lgr->llc_ctl[initiator].qentry = qentry;
+		wake_up_interruptible(&lgr->llc_waiter);
+		break;
+	case LLC_CONFIRM_LINK:
+	case LLC_ADD_LINK_CONT:
+		if (lgr->llc_ctl[LLC_SERV].active) {
+			/* some process is waiting for this response */
+			srv_ctl->qentry = qentry;
+			wake_up_interruptible(&lgr->llc_waiter);
+		}
+		break;
+	case LLC_TEST_LINK:
+		qentry->msg.hd.flags |= LLC_FLAG_RESP;
+		llc_send_msg(qentry->link, &qentry->msg, LLC_NO_NOTIFY);
+		kfree(qentry);
+		break;
+	case LLC_NWM_DATA:
+		lgr->nwm_flags = qentry->msg.hd.flags;
+		if (qentry->msg.hd.flags & LLC_FLAG_HOSTNAME)
+			strncpy(lgr->nwm_data, qentry->msg.data,
+				sizeof(lgr->nwm_data));
+		else
+			memcpy(lgr->nwm_data, qentry->msg.data,
+			       sizeof(lgr->nwm_data));
+		kfree(qentry);
+		break;
+	default:
+		if (!(qentry->msg.hd.type & LLC_OPT_MSG_CTRL)) {
+			/* LLC protocol violation */
+			llc_send_del_link(lnk, NULL, 0,
+					  LLC_DEL_PROT_VIOL, LLC_NO_NOTIFY);
+			smc_terminate_conn(lgr);
+		}
+		kfree(qentry);
+		break;
+	}
+}
+
+void llc_event_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_event_work);
+	struct llc_qentry *qentry;
+
+	if ((!lgr->llc_ctl[LLC_SERV].active) &&
+	    (lgr->delayed_q) &&
+		(atomic_read(&lgr->delayed_q->link->state) != SMC_LINK_FREED))
+		llc_handle_event(lgr->delayed_q, lgr);
+
+again:
+	spin_lock_bh(&lgr->llc_event_q_lock);
+	if (!list_empty(&lgr->llc_event_q)) {
+		qentry = list_first_entry(&lgr->llc_event_q, struct llc_qentry,
+					  list);
+		list_del_init(&qentry->list);
+		spin_unlock_bh(&lgr->llc_event_q_lock);
+		llc_handle_event(qentry, lgr);
+		goto again;
+	}
+	spin_unlock_bh(&lgr->llc_event_q_lock);
+}
+
diff --git a/net/smc/smc_llc.h b/net/smc/smc_llc.h
new file mode 100644
index 0000000..f47c499
--- /dev/null
+++ b/net/smc/smc_llc.h
@@ -0,0 +1,192 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Definitions for LLC message handling
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Klaus Wacker <Klaus.Wacker@de.ibm.com>
+ *              Ursula Braun <Ursula Braun@de.ibm.com>
+ */
+
+#define LLC_DATA_LEN    40
+#define LLC_SEND_POSTED		0xffffffff
+
+#define LLC_FLAG_RESP		0x80
+#define LLC_FLAG_HOSTNAME	0x80
+#define LLC_FLAG_RKEY_DEL	0x40
+#define LLC_FLAG_RKEY_NEG	0x20
+#define LLC_FLAG_RKEY_RETRY	0x10
+#define LLC_FLAG_DEL_ORDERLY	0x20
+#define LLC_FLAG_DEL_ALL	0x40
+#define LLC_FLAG_ADD_LNK_REJ	0x40
+#define LLC_FLAG_NO_ALT_PATH	0x01
+#define LLC_FLAG_LOCAL		0x04    /* Linux-only enhancement */
+#define LLC_FLAGS_DEL_ALL_ORD	0x60
+
+/* LLC DELETE LINK Request Reason Codes */
+#define LLC_DEL_LOST_PATH	0x00010000
+#define LLC_DEL_OP_INIT_TERM	0x00020000
+#define LLC_DEL_PROG_INIT_TERM	0x00030000
+#define LLC_DEL_PROT_VIOL	0x00040000
+#define LLC_DEL_NO_ASYM_NEEDED	0x00050000
+
+enum llc_reqresp {
+	LLC_REQ,
+	LLC_RESP
+};
+
+enum llc_msg_type {
+	LLC_CONFIRM_LINK		= 0x01,
+	LLC_ADD_LINK			= 0x02,
+	LLC_ADD_LINK_CONT		= 0x03,
+	LLC_DEL_LINK			= 0x04,
+	LLC_CONFIRM_RKEY		= 0x06,
+	LLC_CONFIRM_RKEY_CONT		= 0x08,
+	LLC_DELETE_RKEY			= 0x09,
+	LLC_TEST_LINK			= 0x07,
+	LLC_OPT_MSG_CTRL		= 0x80,
+	LLC_NWM_DATA			= 0x8A,
+	RMBE_CTRL			= 0xFE,
+};
+
+struct rmb_rtoken {		/* RMB credentials given to peer */
+	__be32 rmb_key;		/* rkey */
+	__be64 rmb_vaddr;	/* RDMA virtual address */
+} __packed;
+
+struct rmb_rtoken2 {
+	__be32 rmb_key;
+	__be32 rmb_key_new;
+	__be64 rmb_vaddr_new;
+} __packed;
+
+struct rmb_rtoken3 {
+	u8   link_id;
+	__be32 rmb_key;
+	__be64 rmb_vaddr;
+} __packed;
+
+struct llc_hdr {
+	u8 type;
+	u8 length;	/* 44 */
+	u8 version;	/* 0x01 */
+	u8 flags;
+} __packed;
+
+struct llc_msg {
+	struct llc_hdr hd;
+	u8 data[LLC_DATA_LEN];
+} __packed;
+
+struct llc_confirm_msg {		/* type 0x01 */
+	struct llc_hdr hd;
+	u8 sender_mac[6];
+	u8 sender_gid[16];
+	u8 sender_qp_num[3];
+	u8 link_num;
+	__be32 link_uid;
+	u8 max_links;
+	u8 reserved[9];
+} __packed;
+
+struct llc_add_link_msg {		/* type 0x02 */
+	struct llc_hdr hd;
+	u8 sender_mac[6];
+	u8 reserved2[2];
+	u8 sender_gid[16];
+	u8 sender_qp_num[3];
+	u8 link_num;
+	u8 flags2;	/* QP mtu */
+	u8 initial_psn[3];
+	u8 reserved[8];
+} __packed;
+
+struct llc_add_link_cont_msg {	/* type 0x03 */
+	struct llc_hdr hd;
+	u8 link_num;
+	u8 num_rkeys;
+	u8 reserved2[2];
+	struct rmb_rtoken2 rt[2];
+	u8 reserved[4];
+} __packed;
+
+struct llc_del_link_msg {		/* type 0x04 */
+	struct llc_hdr hd;
+	u8 link_num;
+	__be32 reason;
+	u8 reserved[35];
+} __packed;
+
+struct llc_confirm_rkey_msg {	/* type 0x05 */
+	struct llc_hdr hd;
+	struct rmb_rtoken3 rt[3];
+	u8 reserved[1];
+} __packed;
+
+struct llc_conf_rkey_cont_msg {	/* type 0x08 */
+	struct llc_hdr hd;
+	u8 num_rkeys;
+	struct rmb_rtoken3 rt[3];
+} __packed;
+
+struct llc_delete_rkey_msg {	/* type 0x09 */
+	struct llc_hdr hd;
+	u8 num_rkeys;
+	u8 err_mask;
+	u8 reserved[2];
+	__be32 rk[8];
+	u8 reserved2[4];
+} __packed;
+
+struct llc_test_link_msg {		/* type 0x07 */
+	struct llc_hdr hd;
+	u8 user_data[16];
+	u8 reserved[24];
+} __packed;
+
+struct llc_nwm_data_msg {		/* type 0x8A */
+	struct llc_hdr hd;
+	u8 ident[16];
+	u8 reserved[24];
+} __packed;
+
+struct llc_add_link_fake_msg {		/* type 0x82 */
+	struct llc_hdr hd;
+	struct smc_ib_device *ibdev;
+	u8 port_num;
+	u8 reserved[31];
+} __packed;
+
+struct llc_qentry {
+	struct list_head list;
+	struct smc_link *link;
+	struct llc_msg msg;
+};
+
+void llc_init_workers(struct smc_link_group *);
+int llc_enqueue(struct smc_link *, struct llc_msg *);
+int llc_initiate(struct smc_link_group *, enum llc_msg_group);
+void llc_stop(struct smc_link_group *, int);
+struct llc_qentry *llc_wait(struct smc_link_group *, enum smc_role, int, u8);
+int llc_send_confirm_link(struct smc_link *, u8 *, union ib_gid *,
+			  enum llc_reqresp, enum llc_ctl_ind);
+int llc_do_confirm_rkey(struct smc_link *, struct rmb_rx_addrs *);
+int llc_send_test_link(struct smc_link *, u8 *);
+int llc_send_add_link(struct smc_link *, struct smc_roce_defs *,
+		      struct smc_link *, enum llc_reqresp, enum llc_ctl_ind);
+int llc_send_del_link(struct smc_link *, struct smc_link *, u8, u32,
+		      enum llc_ctl_ind);
+int llc_do_del_rkey(struct smc_link *, struct rmb_rx_addrs *);
+void llc_event_worker(struct work_struct *);
+void llc_process_add_link(struct work_struct *);
+void llc_process_del_link(struct work_struct *);
+void llc_process_confirm_rkey(struct work_struct *);
+void llc_process_delete_rkey(struct work_struct *);
+int llc_get_fail_cause(struct smc_link_group *, enum llc_ctl_ind,
+		       struct smc_sock *);
+int llc_cli_add_link(struct smc_link *, struct llc_qentry *, struct smc_sock *);
+int llc_srv_add_link(struct smc_link *, struct smc_roce_defs *,
+		     struct smc_sock *);
+void llc_kill_waiters(struct smc_link_group *);
+void llc_cancel_wait(struct smc_link *);
diff --git a/net/smc/smc_proc.c b/net/smc/smc_proc.c
new file mode 100644
index 0000000..70b94c2
--- /dev/null
+++ b/net/smc/smc_proc.c
@@ -0,0 +1,953 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  auxiliary functions
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              based on prototype from Frank Blaschka
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/seq_file.h>
+#include <linux/debugfs.h>
+#include <linux/uaccess.h>
+#include <linux/workqueue.h>
+#include <linux/proc_fs.h>
+#include <linux/netdevice.h>
+#include <linux/if_vlan.h>
+#include <linux/inetdevice.h>
+#include <linux/types.h>
+#include <net/sock.h>
+#include <net/inet_sock.h>
+#include <net/tcp.h>
+#include <net/route.h>
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <rdma/ib_addr.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+struct smc_pnet_entry {
+	struct list_head list;
+	char dev_name[IFNAMSIZ];
+	char net_name[SMC_MAX_PNET_ID_LEN];
+	u8 port;
+	enum {
+		PNET_ETH,
+		PNET_IB
+	} type;
+	__be32 ipnet;			/* IPv4 address network part */
+	__be32 ipaddr;			/* IPv4 address */
+};
+
+static struct proc_dir_entry *procfs_root;
+static struct proc_dir_entry *procfs_pnet_conf;
+static struct proc_dir_entry *procfs_buff_cnt;
+static struct proc_dir_entry *procfs_max_conn_per_lgr;
+static struct proc_dir_entry *procfs_port_add_delay;
+static struct proc_dir_entry *procfs_sndbuf;
+static struct proc_dir_entry *procfs_rcvbuf;
+static struct list_head smc_pnet_list;
+static rwlock_t smc_pnet_list_lock;
+
+/* determine first gid of roce device  and derive mac from it */
+int smc_get_ib_mac(struct smc_roce_defs *rocdefs)
+{
+	int rc = 0;
+
+	rc = ib_query_gid(rocdefs->ibdev->dev, rocdefs->port, 0, &rocdefs->gid);
+	if (!rc) {
+		memcpy(&rocdefs->mac[0], &rocdefs->gid.raw[8], 3);
+		memcpy(&rocdefs->mac[3], &rocdefs->gid.raw[13], 3);
+		rocdefs->mac[0] &= ~0x02;
+	}
+	return rc;
+}
+
+int smc_port_active(struct smc_ib_device *dev, u8 port)
+{
+	struct ib_port_attr props;
+	int rc;
+
+	rc = ib_query_port(dev->dev, port, &props);
+	if (rc)
+		return 0;
+	if (props.state != IB_PORT_ACTIVE)
+		return 0;
+	return 1;
+}
+
+static void smc_ib_dev_by_pnet(struct smc_roce_defs *rocdefs, char *pnet,
+			       struct smc_link *link)
+{
+	struct smc_ib_device *dev = NULL;
+	struct smc_pnet_entry *entry;
+	int found = 0;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if ((entry->type != PNET_IB) ||
+		    strncmp(pnet, entry->net_name, sizeof(entry->net_name)))
+			continue;
+		if (link &&
+		    !strncmp(entry->dev_name,
+			     link->roce_defs.ibdev->dev->name,
+			     sizeof(entry->dev_name)) &&
+		    (entry->port == link->roce_defs.port)) {
+			continue;
+		}
+		list_for_each_entry(dev, &smc_ib_devices, list) {
+			if (!strncmp(dev->dev->name, entry->dev_name,
+				     sizeof(entry->dev_name))) {
+				read_unlock(&smc_pnet_list_lock);
+				found = smc_port_active(dev, entry->port);
+				read_lock(&smc_pnet_list_lock);
+				if (found)
+					break;
+			}
+		}
+		if (found)
+			break;
+	}
+	read_unlock(&smc_pnet_list_lock);
+
+	if (!found)
+		return;
+
+	rocdefs->ibdev = dev;
+	rocdefs->port = entry->port;
+}
+
+int smc_pnet_by_ibdev(u8 port, char *pnet, struct smc_ib_device *dev)
+{
+	struct smc_pnet_entry *entry;
+	int rc = -EINVAL;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_IB &&
+		    !strncmp(dev->dev->name, entry->dev_name,
+			     sizeof(entry->dev_name)) &&
+		    (port == entry->port)) {
+			rc = 0;
+			strncpy(pnet, entry->net_name, sizeof(entry->net_name));
+			break;
+		}
+	}
+	read_unlock(&smc_pnet_list_lock);
+	return rc;
+}
+
+/* Scan the PNET list and find all entries with different PNET name but
+ * same IP address network part.
+ * Return -EINVAL when identical network part and different PNET ids.
+ */
+static int smc_pnet_checkipnet(struct smc_pnet_entry *hit)
+{
+	int rc = 0;
+	struct smc_pnet_entry *entry;
+
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type != PNET_ETH || hit == entry)
+			continue;
+		if (hit->ipnet == entry->ipnet &&
+		    strcmp(hit->net_name, entry->net_name))
+			rc = -EINVAL;
+	}
+	return rc;
+}
+
+/* Get first primary address of interface listed in PNET table.
+ * Returns number of primary interface addresses detected.
+ */
+static int smc_pnet_get1st(struct smc_pnet_entry *entry, struct net_device *dev)
+{
+	int cnt = 0;
+
+	for_primary_ifa(dev->ip_ptr) {
+		if (!entry->ipaddr) {
+			entry->ipnet = ifa->ifa_address & ifa->ifa_mask;
+			entry->ipaddr = ifa->ifa_address;
+		}
+		++cnt;
+	} endfor_ifa(dev->ip_ptr);
+	return cnt;
+}
+
+/* Get net_device assigned to an interface and find the first IP address and
+ * IP network address.
+ *
+ * Note:
+ * Use the same network namespace as the incoming (destination) device. This
+ * should be identical to the process network namespace.
+ */
+static void smc_pnet_getipdata(struct smc_pnet_entry *entry, struct net *ns)
+{
+	struct net_device *dev = dev_get_by_name(ns, entry->dev_name);
+
+	entry->ipnet = 0;
+	entry->ipaddr = 0;
+	if (dev) {
+		smc_pnet_get1st(entry, dev);
+		dev_put(dev);
+	}
+}
+
+static int smc_pnet_by_tcpsk(char *pnet, struct sock *sk)
+{
+	int found = -EINVAL;
+	struct smc_pnet_entry *entry, *hit = NULL;
+	struct dst_entry *dst = sk_dst_get(sk);
+
+	if (!dst)
+		goto out;
+	if (!dst->dev)
+		goto out_rel;
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_ETH) {
+			smc_pnet_getipdata(entry, dev_net(dst->dev));
+			if (found == -EINVAL &&
+			    !strncmp(dst->dev->name, entry->dev_name,
+				     sizeof(entry->dev_name))) {
+				strncpy(pnet, entry->net_name,
+					sizeof(entry->net_name));
+				found = 0;
+				hit = entry;
+			}
+		}
+	}
+	if (hit)
+		found = smc_pnet_checkipnet(hit);
+	read_unlock(&smc_pnet_list_lock);
+out_rel:
+	dst_release(dst);
+out:
+	return found;
+}
+
+int smc_gid_by_dev(struct smc_roce_defs *rocdefs)
+{
+	int rc = 0;
+
+	rc = ib_query_gid(rocdefs->ibdev->dev, rocdefs->port, 0,
+			  &rocdefs->gid);
+	rocdefs->sgid_idx = 0;
+	smc_get_ib_mac(rocdefs);
+	return rc;
+}
+
+static int smc_gid_by_tcpsk(struct smc_roce_defs *rocdefs, struct sock *sk)
+{
+	struct dst_entry *dst = sk_dst_get(sk);
+	int rc;
+
+	rocdefs->vlan = 0;
+	if (dst) {
+		if (dst->dev) {
+			if (is_vlan_dev(dst->dev))
+				rocdefs->vlan = vlan_dev_vlan_id(dst->dev);
+		}
+		dst_release(dst);
+	}
+	rc = smc_gid_by_dev(rocdefs);
+	return rc;
+}
+
+void smc_find_roce_resources(struct smc_roce_defs *rocdefs, char *pnet,
+			     struct sockaddr_in *inaddr, struct sock *sk)
+{
+	int rc;
+
+	rocdefs->ibdev = NULL;
+	rocdefs->port = 0;
+	rc = smc_pnet_by_tcpsk(pnet, sk);
+	if (rc) {
+		/* add fallback path and try to find ib dev/port
+		 * via mac address of tcp sock
+		 */
+		return;
+	}
+	smc_ib_dev_by_pnet(rocdefs, pnet, NULL);
+	if (!rocdefs->ibdev)
+		return;
+
+	rc = smc_gid_by_tcpsk(rocdefs, sk);
+	if (rc) {
+		rocdefs->ibdev = NULL;
+		return;
+	}
+}
+
+void smc_find_alt_roce_resources(struct smc_roce_defs *rocdefs,
+				 struct smc_link_group *lgr,
+				 struct smc_link *link)
+{
+	int rc;
+
+	smc_ib_dev_by_pnet(rocdefs, lgr->pnet_id, link);
+	if (!rocdefs->ibdev)
+		return;
+
+	rocdefs->vlan = lgr->vlan;
+	rc = smc_gid_by_dev(rocdefs);
+	if (rc) {
+			rocdefs->ibdev = NULL;
+			return;
+	}
+}
+
+int smc_netinfo_by_tcpsk(__be32 *subnet, u8 *mlen, struct socket *tcpsocket)
+{
+	struct dst_entry *dst = sk_dst_get(tcpsocket->sk);
+	struct sockaddr_in addr;
+	int rc = -ENODEV;
+	int len;
+
+	if (!dst)
+		goto out;
+	if (!dst->dev)
+		goto out_rel;
+	kernel_getsockname(tcpsocket, (struct sockaddr *)&addr, &len);
+	for_ifa(dst->dev->ip_ptr) {
+		if (ifa->ifa_address != addr.sin_addr.s_addr)
+			continue;
+		*mlen = inet_mask_len(ifa->ifa_mask);
+		*subnet = ifa->ifa_address & ifa->ifa_mask;
+		rc = 0;
+		break;
+	} endfor_ifa(dst->dev->ip_ptr);
+out_rel:
+	dst_release(dst);
+out:
+	return rc;
+}
+
+static void smc_print_area(struct seq_file *m, char *buf, int max_len,
+			   int pr_len)
+{
+	char fmtbuf[60];
+	int i, len;
+
+	len = min_t(size_t, max_len, pr_len);
+	while (len > 0) {
+		i = (len < 16) ? len : 16;
+		hex_dump_to_buffer(buf, i, 16, 4, fmtbuf, 60, 0);
+		seq_printf(m, "%p: %s\n", buf, fmtbuf);
+		buf += 16;
+		len -= 16;
+	}
+}
+
+static int smc_sock_show(struct seq_file *m, void *v)
+{
+	struct smc_proc_dir_entry *proc = m->private;
+	struct sock *sk;
+	struct smc_sock *smc;
+	struct sockaddr_in locl_addr, peer_addr;
+	int rc, len;
+
+	if (!proc)
+		return 0;
+
+	sk = proc->sk;
+	if (!sk)
+		return 0;
+
+	sock_hold(sk);
+	if (sock_flag(sk, SOCK_DEAD) || (sk->sk_state == SMC_DESTRUCT))
+		goto out;
+
+	smc = smc_sk(sk);
+	seq_printf(m, "state: %d\n", sk->sk_state);
+	seq_printf(m, "uid: %d\n",
+		   from_kuid_munged(seq_user_ns(m), sock_i_uid(sk)));
+	seq_printf(m, "inode: %ld\n", sock_i_ino(sk));
+
+	if (sk->sk_state == SMC_INIT)
+		goto out;
+
+	if ((smc->tcpsocket) && (smc->tcpsocket->sk)) {
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket,
+						  (struct sockaddr *)&locl_addr,
+						  &len, 0);
+		if (!rc) {
+			seq_printf(m, "local_addr: %pI4\n",
+				   &locl_addr.sin_addr.s_addr);
+			seq_printf(m, "local_port: %d\n",
+				   ntohs(locl_addr.sin_port));
+		}
+	}
+
+	if (sk->sk_state == SMC_LISTEN)
+		goto out;
+
+	if ((smc->tcpsocket) && (smc->tcpsocket->sk)) {
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket,
+						  (struct sockaddr *)&peer_addr,
+						  &len, 1);
+		if (!rc) {
+			seq_printf(m, "peer_addr: %pI4\n",
+				   &peer_addr.sin_addr.s_addr);
+			seq_printf(m, "peer_port: %d\n",
+				   ntohs(peer_addr.sin_port));
+		}
+	}
+
+	seq_printf(m, "tcp_fallback: %d\n", smc->use_tcp);
+	if (smc->use_tcp)
+		goto out;
+
+	if (smc->conn.lgr && (sk->sk_state != SMC_CLOSED)) {
+		seq_printf(m, "link group target: %#x\n",
+			   smc->conn.lgr->daddr);
+		seq_printf(m, "link group role: %d\n", smc->conn.lgr->role);
+		seq_puts(m, "gid_peer:\n");
+		smc_print_area(m, smc->conn.lgr->lnk[0].gid_peer, 16, 16);
+		smc_print_area(m, smc->conn.lgr->lnk[1].gid_peer, 16, 16);
+	}
+	seq_printf(m, "tx buffer len: %d\n", smc->conn.rmb_tx_size);
+	seq_printf(m, "rx buffer len: %d\n", smc->conn.rmb_rx_size);
+	seq_printf(m, "local alert token: %#x\n", smc->conn.alert_token_local);
+
+	seq_printf(m, "peer rx buffer len: %d\n", smc->conn.peer_rx_buf_len);
+	seq_printf(m, "local rx pc %d pw %d cc %d cw %d\n",
+		   smc->conn.local_rx_ctrl.p_curs.s.curs.c,
+		   smc->conn.local_rx_ctrl.p_curs.s.curs.w,
+		   smc->conn.local_rx_ctrl.c_curs.s.curs.c,
+		   smc->conn.local_rx_ctrl.c_curs.s.curs.w);
+	seq_printf(m, "local tx pc %d pw %d cc %d cw %d\n",
+		   smc->conn.local_tx_ctrl.p_curs.s.curs.c,
+		   smc->conn.local_tx_ctrl.p_curs.s.curs.w,
+		   smc->conn.local_tx_ctrl.c_curs.s.curs.c,
+		   smc->conn.local_tx_ctrl.c_curs.s.curs.w);
+	seq_printf(m, "tx producer_flags %#x conn_state_flags %#x\n",
+		   *(u8 *)&smc->conn.local_tx_ctrl.p_flags,
+		   *(u8 *)&smc->conn.local_tx_ctrl.conn_state_flags);
+	seq_printf(m, "rx producer_flags %#x conn_state_flags %#x\n",
+		   *(u8 *)&smc->conn.local_rx_ctrl.p_flags,
+		   *(u8 *)&smc->conn.local_rx_ctrl.conn_state_flags);
+
+	if (smc->conn.rmb_tx_elem &&
+	    smc->conn.rmb_tx_elem->rmb_tx &&
+	    smc->conn.rmb_tx_elem->rmb_tx) {
+		seq_puts(m, "tx buffer:\n");
+		smc_print_area(m, smc->conn.rmb_tx_elem->rmb_tx/*->buffer*/,
+			       smc->conn.rmb_tx_size, 15000);
+	}
+	if (smc->conn.rmb_rx_elem &&
+	    smc->conn.rmb_rx_elem->rmb_rx &&
+	    smc->conn.rmb_rx_elem->rmb_rx) {
+		seq_puts(m, "rx buffer:\n");
+		smc_print_area(m, smc->conn.rmb_rx_elem->rmb_rx/*->buffer*/,
+			       smc->conn.rmb_rx_size, 15000);
+	}
+
+out:
+	sock_put(sk);
+	return 0;
+}
+
+static int smc_sock_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_sock_show, PDE_DATA(inode));
+}
+
+static const struct file_operations procfs_smc_sock_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_sock_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.release = single_release,
+};
+
+int smc_sock_proc_create(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+
+	sprintf(smc->proc_name, "Sx%p_0x%p", sk, smc->proc);
+	smc->proc->proc = proc_create_data(smc->proc_name,
+					   S_IFREG | S_IRUGO | S_IWUSR,
+					   procfs_root, &procfs_smc_sock_fops,
+					   smc->proc);
+	return 0;
+}
+
+static int smc_pnet_show(struct seq_file *m, void *v)
+{
+	struct smc_pnet_entry *entry;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_IB)
+			seq_printf(m, "%-*s ib  %s %d\n", SMC_MAX_PNET_ID_LEN,
+				   entry->net_name, entry->dev_name,
+				   entry->port);
+		else
+			seq_printf(m, "%-*s eth %s\n", SMC_MAX_PNET_ID_LEN,
+				   entry->net_name, entry->dev_name);
+	}
+	read_unlock(&smc_pnet_list_lock);
+	return 0;
+}
+
+static struct smc_pnet_entry *smc_pnet_create_entry(int argc, char *argv[])
+{
+	struct smc_pnet_entry *entry;
+	long rc = 0;
+
+	entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+	if (!entry) {
+		rc = -ENOMEM;
+		goto out;
+	}
+	rc = -EINVAL;
+	if (!strcmp(argv[2], "ib")) {
+		if ((argc != 5) || kstrtou8(argv[4], 0, &entry->port))
+			goto out;
+		entry->type = PNET_IB;
+	} else if (!strcmp(argv[2], "eth")) {
+		if (argc > 4)
+			goto out;
+		entry->type = PNET_ETH;
+	} else {
+		goto out;
+	}
+	if (strlen(argv[1]) >= sizeof(entry->net_name) ||
+	    strlen(argv[3]) >= sizeof(entry->dev_name))
+		goto out;
+	strncpy(entry->net_name, argv[1], sizeof(entry->net_name));
+	strncpy(entry->dev_name, argv[3], sizeof(entry->dev_name));
+	return entry;
+
+out:
+	kfree(entry);
+	return ERR_PTR(rc);
+}
+
+/* Update the PNET ID table. Make sure entries are sorted by PNET ID
+ * for fancy output.
+ * @entry: entry to search for
+ * @del: delete existing entry if found, otherwise add.
+ *
+ * Returns 0 on success, <0 otherwise.
+ */
+static int smc_pnet_update(struct smc_pnet_entry *entry, bool del)
+{
+	struct smc_pnet_entry *tmp_entry, *n_entry;
+	int rc = -EINVAL;
+	int found = 0;
+
+	/* keep list sorted by PNET ID */
+	write_lock(&smc_pnet_list_lock);
+	list_for_each_entry_safe(tmp_entry, n_entry, &smc_pnet_list, list) {
+		if (entry->type == tmp_entry->type &&
+		    !strncmp(entry->dev_name, tmp_entry->dev_name,
+		    sizeof(entry->dev_name)) &&
+		    entry->port == tmp_entry->port) {
+			/* entry for this device, port and type found */
+			if (del) {
+				list_del(&tmp_entry->list);
+				kfree(tmp_entry);
+				kfree(entry);
+				rc = 0;
+			}
+			found = 1;
+			break;
+		}
+		if (strncmp(entry->net_name, tmp_entry->net_name,
+			    sizeof(entry->net_name)) < 0)
+			break;
+	}
+	if (!del && !found) {
+		rc = 0;
+		if (entry->type == PNET_ETH) {
+			smc_pnet_getipdata(entry, current->nsproxy->net_ns);
+			rc = smc_pnet_checkipnet(entry);
+		}
+		if (!rc)
+			list_add_tail(&entry->list, &tmp_entry->list);
+	}
+	write_unlock(&smc_pnet_list_lock);
+
+	return rc;
+}
+
+/* add|del <PNET ID> eth|ib <device name> [<port>] */
+static ssize_t smc_pnet_write(struct file *file,
+			      const char __user *user_buffer,
+			      size_t count, loff_t *offset)
+{
+	char buf[128];
+	char *pos;
+	char *tok;
+	char *tmp;
+	char *argv[5];
+	int argc, rc = -EINVAL, del;
+	size_t len = min_t(size_t, count, sizeof(buf) - 1);
+	struct smc_pnet_entry *entry;
+
+	memset(buf, 0, sizeof(buf));
+	argc = 0;
+
+	if (copy_from_user(buf, user_buffer, len)) {
+		rc = -EFAULT;
+		goto out;
+	}
+	tmp = buf;
+	pos = strsep(&tmp, "\n");
+
+	while ((tok = strsep(&pos, " "))) {
+		if (!strcmp(tok, ""))
+			continue;
+		if (argc == 5)
+			break;
+		argv[argc] = tok;
+		argc++;
+	}
+
+	if (argc < 4 || tok)
+		goto out;
+
+	if (!strcmp(argv[0], "add"))
+		del = 0;
+	else if (!strcmp(argv[0], "del"))
+		del = 1;
+	else
+		goto out;
+	entry = smc_pnet_create_entry(argc, argv);
+	if (IS_ERR(entry)) {
+		rc = PTR_ERR(entry);
+		goto out;
+	}
+	rc = smc_pnet_update(entry, del);
+	if (!rc)
+		rc = count;
+out:
+	return rc;
+}
+
+static int smc_pnet_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_pnet_show, PDE_DATA(inode));
+}
+
+static int smc_buff_cnt_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%u\n", smc_ctrl_buf_cnt);
+	return 0;
+}
+
+static ssize_t smc_buff_cnt_write(struct file *file,
+				  const char __user *user_buffer,
+				  size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 3) ||
+	    (tmp > SMC_MAX_WRE) ||
+	    ((smc_max_conn_per_lgr * 10 * tmp) > SMC_MAX_CQE)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	spin_lock_bh(&smc_lgr_list.lock);
+	if (!list_empty(&smc_lgr_list.list)) {
+		rc = -EPERM;
+		goto out_unlock;
+	}
+	smc_ctrl_buf_cnt = tmp;
+	rc = count;
+out_unlock:
+	spin_unlock_bh(&smc_lgr_list.lock);
+out:
+	return rc;
+}
+
+static int smc_buff_cnt_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_buff_cnt_show, PDE_DATA(inode));
+}
+
+static int smc_max_conn_per_lgr_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_max_conn_per_lgr);
+	return 0;
+}
+
+static ssize_t smc_max_conn_per_lgr_write(struct file *file,
+					  const char __user *user_buffer,
+					  size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 1) ||
+	    (tmp > SMC_MAX_RMB) ||
+	    ((tmp * 10 * smc_ctrl_buf_cnt) > SMC_MAX_CQE)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_max_conn_per_lgr = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_max_conn_per_lgr_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_max_conn_per_lgr_show, PDE_DATA(inode));
+}
+
+static int smc_port_add_delay_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_port_add_delay);
+	return 0;
+}
+
+static ssize_t smc_port_add_delay_write(struct file *file,
+					const char __user *user_buffer,
+					size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 0) ||
+	    (tmp > SMC_MAX_PORT_ADD_DELAY)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_port_add_delay = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_port_add_delay_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_port_add_delay_show, PDE_DATA(inode));
+}
+
+static int smc_sndbuf_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_def_sndbuf_size);
+	return 0;
+}
+
+static ssize_t smc_sndbuf_write(struct file *file,
+				const char __user *user_buffer, size_t count,
+				loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < SK_MEM_QUANTUM) ||
+	    (tmp > sysctl_wmem_max)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_def_sndbuf_size = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_sndbuf_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_sndbuf_show, PDE_DATA(inode));
+}
+
+static int smc_rcvbuf_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_def_rcvbuf_size);
+	return 0;
+}
+
+static ssize_t smc_rcvbuf_write(struct file *file,
+				const char __user *user_buffer,
+				size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < SK_MEM_QUANTUM) ||
+	    (tmp > sysctl_rmem_max)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_def_rcvbuf_size = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_rcvbuf_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_rcvbuf_show, PDE_DATA(inode));
+}
+
+static const struct file_operations procfs_pnet_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_pnet_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_pnet_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_buff_cnt_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_buff_cnt_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_buff_cnt_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_max_conn_per_lgr_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_max_conn_per_lgr_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_max_conn_per_lgr_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_port_add_delay_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_port_add_delay_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_port_add_delay_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_sndbuf_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_sndbuf_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_sndbuf_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_rcvbuf_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_rcvbuf_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_rcvbuf_write,
+	.release = single_release,
+};
+
+int smc_proc_init(void)
+{
+	INIT_LIST_HEAD(&smc_pnet_list);
+	rwlock_init(&smc_pnet_list_lock);
+	procfs_root = proc_net_mkdir(&init_net, "smc", init_net.proc_net);
+	if (!procfs_root)
+		goto out;
+	procfs_pnet_conf =
+		proc_create_data("pnet_conf", S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_pnet_fops, NULL);
+	if (!procfs_pnet_conf)
+		goto out_smc;
+	procfs_buff_cnt =
+		proc_create_data("ctrl_buffer_count",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_buff_cnt_fops, NULL);
+	if (!procfs_buff_cnt)
+		goto out_pnet_conf;
+	procfs_max_conn_per_lgr =
+		proc_create_data("max_conn_per_lgr",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_max_conn_per_lgr_fops,
+				 NULL);
+	if (!procfs_max_conn_per_lgr)
+		goto out_ctrl_buffer_count;
+	procfs_port_add_delay =
+		proc_create_data("port_add_delay",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_port_add_delay_fops,
+				 NULL);
+	if (!procfs_port_add_delay)
+		goto out_max_conn_per_lgr;
+	procfs_sndbuf = proc_create_data("sndbuf", S_IFREG | S_IRUGO | S_IWUSR,
+					 procfs_root, &procfs_sndbuf_fops,
+					 NULL);
+	if (!procfs_sndbuf)
+		goto out_port_add_delay;
+	procfs_rcvbuf = proc_create_data("rcvbuf", S_IFREG | S_IRUGO | S_IWUSR,
+					 procfs_root, &procfs_rcvbuf_fops,
+					 NULL);
+	if (!procfs_rcvbuf)
+		goto out_sndbuf;
+	return 0;
+
+out_sndbuf:
+	remove_proc_entry("sndbuf", procfs_root);
+out_port_add_delay:
+	remove_proc_entry("port_add_delay", procfs_root);
+out_max_conn_per_lgr:
+	remove_proc_entry("max_conn_per_lgr", procfs_root);
+out_ctrl_buffer_count:
+	remove_proc_entry("ctrl_buffer_count", procfs_root);
+out_pnet_conf:
+	remove_proc_entry("pnet_conf", procfs_root);
+out_smc:
+	remove_proc_entry("smc", init_net.proc_net);
+out:
+	return -EFAULT;
+}
+
+void smc_proc_exit(void)
+{
+	struct smc_pnet_entry *entry, *tmp_entry;
+
+	write_lock(&smc_pnet_list_lock);
+	list_for_each_entry_safe(entry, tmp_entry, &smc_pnet_list, list) {
+		list_del(&entry->list);
+		kfree(entry);
+	}
+	write_unlock(&smc_pnet_list_lock);
+
+	if (procfs_rcvbuf)
+		remove_proc_entry("rcvbuf", procfs_root);
+	if (procfs_sndbuf)
+		remove_proc_entry("sndbuf", procfs_root);
+	if (procfs_port_add_delay)
+		remove_proc_entry("port_add_delay", procfs_root);
+	if (procfs_max_conn_per_lgr)
+		remove_proc_entry("max_conn_per_lgr", procfs_root);
+	if (procfs_buff_cnt)
+		remove_proc_entry("ctrl_buffer_count", procfs_root);
+	if (procfs_pnet_conf)
+		remove_proc_entry("pnet_conf", procfs_root);
+	if (procfs_root)
+		remove_proc_entry("smc", init_net.proc_net);
+}
-- 
2.3.8

[PATCH V3 net-next 5/5] smc: increase / decrease static key

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

To activate the smc-specific hooks in the TCP-code, static key "tcp_have_smc"
is increased and decreased. For now this is done at module load / unload time.

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 net/smc/af_smc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index ed62370..8256e17 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -3142,6 +3142,7 @@ static int __init af_smc_init(void)
 	if (rc)
 		goto out_sock;
 
+	static_key_slow_inc(&tcp_have_smc);
 	return 0;
 
 out_sock:
@@ -3154,6 +3155,7 @@ out:
 
 static void __exit af_smc_exit(void)
 {
+	static_key_slow_dec(&tcp_have_smc);
 	ib_unregister_client(&smc_ib_client);
 	sock_unregister(PF_SMC);
 	proto_unregister(&smc_proto);
-- 
2.3.8

Re: [PATCH V3 net-next 0/5] net: implement SMC-R solution

[David Miller]

From: Ursula Braun <ubraun@linux.vnet.ibm.com>
Date: Wed, 22 Jul 2015 10:59:47 +0200

> 1. Provides optimized performance compared to standard TCP/IP over Ethernet
>    within the data center for both request/response (latency) and streaming
>    workloads (CPU savings) [3]. 
>    Initial benchmarks on Linux on x86 processors have shown latency
>    reduction of up to 52% with a throughput gain of 111% using SMC-R vs TCP
>    for request/response message patterns (10 concurrent TCP connections
>    with 16KB    messages) and CPU savings of up to 69% for streaming data
>    patterns (single TCP connection with 20MB of data in one direction).
>    [1] is currently updated to contain more detailed information on Linux
>    and performance.

I'm really sorry but this is the same rabbit hole and set of claims
that have been bullhorned my way for RDMA over the years and I still
don't buy it.

None of the RDMA'ish proponents ever talk about what you _don't_ get
when this stuff triggers.

No netfilter.

No packet scheduler.

No classifier actions.

No BPF.

No bridging.

Basically, every single interesting feature of the Linux networking
goes away once this RDMA thing happens.

Furthermore the benchmarks are carefully choosen to exemplify the
perfect environment for this feature to excell at.

Sorry, I don't want any of this in our core networking stack.  You'll
have to support this completely outside of the TCP implementation and
core networking code, and therefore have it %100 in your own separate
module with your own can of worms like the Infiniband et al. people
do.

Having this standardized or "in widespread successful use" has no
bearing upon the things I do not like about this patch set, so do not
use those kinds of arguments to try and change my mind.  It won't
work.

I'm not applying this patch series.

Re: [PATCH V3 net-next 0/5] net: implement SMC-R solution

[Ursula Braun]

On Sun, 2015-07-26 at 16:15 -0700, David Miller wrote:

> I'm really sorry but this is the same rabbit hole and set of claims
> that have been bullhorned my way for RDMA over the years and I still
> don't buy it.
> 
> None of the RDMA'ish proponents ever talk about what you _don't_ get
> when this stuff triggers.
> 
> No netfilter.
> 
> No packet scheduler.
> 
> No classifier actions.
> 
> No BPF.
> 
> No bridging.
> 
> Basically, every single interesting feature of the Linux networking
> goes away once this RDMA thing happens.
Correct, Dave: All the RDMA protocols do not benefit from packet-based
features of Linux networking. Which is why SMC-R starts with TCP to
maintain some of it. This allows for adhering to the TCP management
model, e.g. for addressing and capability to intercept connection
establishment. We did SMC-R for a good reason -- bringing value to the
mainframe datacenter environment, and potentially other setups, too; and
that includes performance benefits.
> 
> Furthermore the benchmarks are carefully choosen to exemplify the
> perfect environment for this feature to excell at.
Our benchmarks shown are by no means "some carefully chosen" selection.
Would you like to see some other benchmarks?
> 
> Sorry, I don't want any of this in our core networking stack.  You'll
> have to support this completely outside of the TCP implementation and
> core networking code, and therefore have it %100 in your own separate
> module with your own can of worms like the Infiniband et al. people
> do.
Since you are asking for a solution "100% in our own separate module
with our own can of worms", we have to give up the transparent detection
whether a communication peer can do SMC-R or not (this has been the
purpose of the rejected TCP hooks). Instead, we want just the new
self-contained SMC-R socket family added to the kernel. We will adapt
the code accordingly to get rid of TCP experimental options. Is it ok
for you, if we send the whole new code for /net/smc/ in one single patch
since all of it is required for doing the job, or prefer smaller chunks
(which, if half way applied, won't get real functionality)?

Kind regards, Ursula

[PATCH V4 net-next 0/2] net: implement SMC-R solution

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

Dave,

this is V4 of my SMC-R patches:
Since you are asking for a solution "100% in our own separate module
with our own can of worms", we have to give up the transparent detection
whether a communication peer can do SMC-R or not (this has been the
purpose of the rejected TCP hooks). Instead, we want just the new
self-contained SMC-R socket family added to the kernel.
By the way, since August 2015 the SMC-R Informational RFC is no longer
a draft, but published as RFC7609.

V4 changes:
1. Remove tcp patches supporting TCP experimental options
2. Remove references to tcp_sock syn_smc flag in smc-code, since TCP
   experimental options are not supported by the Linux-tcp.
3. clc_wait_msg() simplified

V3 changes:
1. Avoid adding of new space for smc-related bits in the tcp structures.
2. Make the smc feature to be nearly zero cost using Static Keys / jump
   labels
3. Increase / decrease smc static key in the smc-code
4. Make sure the next-to-last patch does not break the build
5. Additional pnet table checking

V2 changes:
1. activate tcp changes for CONFIG_AFSMC only (as suggested by Eric Dumazet)
2. add additional hook in net/core/sock.c
3. fix bitfield endianness problem

Thanks,
        Ursula

In 2013, IBM introduced an optimized communications solution for the
IBM zEnterprise EC12 and BC12 (s390 in Linux terminology) that is
comprised of the IBM 10GbE RoCE Express feature with Shared Memory
Communications-RDMA (SMC-R) protocol [1].
SMC-R is designed for the enterprise data center environment and is an open
protocol as specified in the informational RFC7609 [2]. It has been
published in August 2015. Another implementation of this protocol is
available since 2013 with IBM z/OS Version 2 Release 1. 

SMC-R provides a “sockets over RDMA” solution that leverages industry
standard RDMA over Converged Ethernet (RoCE) technology.

IBM has developed a Linux implementation of the SMC-R standard. A new
socket protocol family AF_SMC is introduced. A preload library can be used
to enable TCP-based applications to use SMC-R without changes. 

Key aspects of SMC-R are: 
1. Provides optimized performance compared to standard TCP/IP over Ethernet
   within the data center for both request/response (latency) and streaming
   workloads (CPU savings) [3]. 
   Initial benchmarks on Linux on x86 processors have shown latency
   reduction of up to 52% with a throughput gain of 111% using SMC-R vs TCP
   for request/response message patterns (10 concurrent TCP connections
   with 16KB    messages) and CPU savings of up to 69% for streaming data
   patterns (single TCP connection with 20MB of data in one direction).
   [1] is currently updated to contain more detailed information on Linux
   and performance.
2. In order to preserve the traditional network administrative model the
   SMC-R protocol ties into the existing IP addresses and uses TCP's
   handshake to establish connections. This allows existing management
   tools and security infrastructure to control the creation of SMC
   connections.
3. The SMC-R protocol logically bonds multiple RoCE adapters together
   providing    redundancy with transparent fail-over for improved high
   availability, increased bandwidth and load balancing across multiple
   RDMA-capable devices.
Without the rejected TCP Experimental Options the following aspects are
restricted; alternate solutions are in discussion. 
4. Due to its handshake protocol, SMC-R is compatible with (transparent to)
   existing TCP connection load balancers that are commonly used in the
   enterprise data center environment for multi-tier application workloads.
5. SMC-R's handshake protocol allows for transparent fallback to TCP/IP,
   should one of the peers not be capable of the protocol.

Additional SMC-R overview and reference materials are available [1].  

The SMC-R “rendezvous" protocol eliminates the need for RDMA-CM and the
exchange occurs through an initial TCP connection. Building on a TCP
connection to establish an SMC-R connection solves many key requirements.
The rendezvous process occurs now in 1 phase only: 
1. TCP/IP 3-way exchange with TCP experimental options is skipped.
2. SMC-R 3-way exchange:
   It is assumed both partners indicate SMC-R capability. Then at the
   completion of the 3-way TCP handshake the SMC-R layers in each peer take
   control of the TCP connection and exchange their RDMA credentials. If
   this 3-way exchange completes successfully the connection continues using
   SMC-R. If the exchange is not successful the connections falls back to
   standard TCP/IP. 

References:
[1] SMC-R Overview and Reference Materials:
    http://www-01.ibm.com/software/network/commserver/SMCR/ 
[2] SMC-R Informational RFC:
    https://tools.ietf.org/rfc/rfc7609
[3] Linux SMC-R Overview and Performance Summary
    (archs x86 and s390):
    http://www-01.ibm.com/software/network/commserver/SMCR/ 

The patch series is prepared to apply to net-next and consists of these
parts:
1. net:             definitions to establish new socket family
2. net/smc:         new socket family

In the future, SMC-R will be enhanced to cover:
- alternate SMC-capability detection
- IPv6 support
- Tracing
- Statistics support

shortlog:

Ursula Braun (2):
  net: introduce socket family constants
  smc: introduce socket family AF_SMC

-- 
2.3.8

[PATCH V4 net-next 1/2] net: introduce socket family constants

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

The new socket family is assigned the next available address / protocol
family constant 41.
Implementing SO_KEEPALIVE for SMC-R requires an extra hook in net/ipv4/timer.c.

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 include/linux/socket.h |  4 +++-
 include/net/smc.h      | 13 +++++++++++++
 net/ipv4/tcp_timer.c   |  2 +-
 3 files changed, 17 insertions(+), 2 deletions(-)
 create mode 100644 include/net/smc.h

diff --git a/include/linux/socket.h b/include/linux/socket.h
index 5bf59c8..1adcbcc 100644
--- a/include/linux/socket.h
+++ b/include/linux/socket.h
@@ -200,7 +200,8 @@ struct ucred {
 #define AF_ALG		38	/* Algorithm sockets		*/
 #define AF_NFC		39	/* NFC sockets			*/
 #define AF_VSOCK	40	/* vSockets			*/
-#define AF_MAX		41	/* For now.. */
+#define AF_SMC		41	/* smc sockets			*/
+#define AF_MAX		42	/* For now.. */
 
 /* Protocol families, same as address families. */
 #define PF_UNSPEC	AF_UNSPEC
@@ -246,6 +247,7 @@ struct ucred {
 #define PF_ALG		AF_ALG
 #define PF_NFC		AF_NFC
 #define PF_VSOCK	AF_VSOCK
+#define PF_SMC		AF_SMC
 #define PF_MAX		AF_MAX
 
 /* Maximum queue length specifiable by listen.  */
diff --git a/include/net/smc.h b/include/net/smc.h
new file mode 100644
index 0000000..cd513ee
--- /dev/null
+++ b/include/net/smc.h
@@ -0,0 +1,13 @@
+/*
+ * SMC		Definitions for the SMC protocol.
+ *
+ * Author:	Ursula Braun <ursula.braun@de.ibm.com>
+ */
+#ifndef _SMC_H
+#define _SMC_H
+
+/* SMC socket options - disjunct with TCP socket options */
+#define SMC_KEEPALIVE		99	/* start/stop keepalives */
+
+#endif /* _SMC_H */
+
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 7149ebc..070bfc7 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -557,7 +557,7 @@ void tcp_set_keepalive(struct sock *sk, int val)
 	else if (!val)
 		inet_csk_delete_keepalive_timer(sk);
 }
-
+EXPORT_SYMBOL(tcp_set_keepalive);
 
 static void tcp_keepalive_timer (unsigned long data)
 {
-- 
2.3.8

[PATCH V4 net-next 2/2] smc: introduce socket family AF_SMC

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

In 2013, IBM introduced an optimized communications solution for the
IBM zEnterprise EC12 and BC12 (s390 in Linux terminology) that is
comprised of the IBM 10GbE RoCE Express feature with Shared Memory
Communications-RDMA (SMC-R) protocol [1].
SMC-R is designed for the enterprise data center environment and is an open
protocol as specified in the informational RFC7609 [2]. It has been
published in August 2015. Another implementation of this protocol is
available since 2013 with IBM z/OS Version 2 Release 1.

SMC-R provides a “sockets over RDMA” solution that leverages industry
standard RDMA over Converged Ethernet (RoCE) technology.

IBM has developed a Linux implementation of the SMC-R standard. A new
socket protocol family AF_SMC is introduced. A preload library can be used
to enable TCP-based applications to use SMC-R without changes.

Key aspects of SMC-R are:
1. Provides optimized performance compared to standard TCP/IP over Ethernet
   within the data center for both request/response (latency) and streaming
   workloads (CPU savings) [3].
2. In order to preserve the traditional network administrative model the
   SMC-R protocol ties into the existing IP addresses and uses TCP's
   handshake to establish connections. This allows existing management
   tools and security infrastructure to control the creation of SMC
   connections.
3. The SMC-R protocol logically bonds multiple RoCE adapters together
   providing redundancy with transparent fail-over for improved high
   availability, increased bandwidth and load balancing across multiple
   RDMA-capable devices.
4. Due to its handshake protocol, SMC-R is compatible with (transparent to)
   existing TCP connection load balancers that are commonly used in the
   enterprise data center environment for multi-tier application workloads.
5. SMC-R's handshake protocol allows for transparent fallback to TCP/IP,
   should one of the peers not be capable of the protocol.

Additional SMC-R overview and reference materials are available [1].

The SMC-R “rendezvous" protocol eliminates the need for RDMA-CM and the
exchange occurs through an initial TCP connection. Building on a TCP
connection to establish an SMC-R connection solves many key requirements.
The rendezvous process occurs now in 1 phase only:
1. TCP/IP 3-way exchange with TCP experimental options is skipped.
2. SMC-R 3-way exchange:
   It is assumed both partners indicate SMC-R capability. Then at the
   completion of the 3-way TCP handshake the SMC-R layers in each peer take
   control of the TCP connection and exchange their RDMA credentials. If
   this 3-way exchange completes successfully the connection continues using
   SMC-R. If the exchange is not successful the connections falls back to
   standard TCP/IP.

A connection using SMC-R starts out setting up a connection using the
TCP/IP protocol [2]. Thus an internal smc-managed TCP socket is
established, whose traffic flows across an interface belonging to the same
Converged Ethernet fabric, defined by a so-called "pnet table" [1].

Since SMC-R negotiation during TCP handshake is no longer possible, it
is assumed both parties are SMC-R capable. The TCP/IP connection is
used to exchange the necessary connection data for setup of RoCE RC-QP
connection in the so-called "rendezvous handshake".

The code implements also the link management of the SMC-R protocol
forming link groups for each remote host and provides for
- dynamic addition of further RoCE Host Channel Adapters (HCAs) to form a
  so-called link group
- seamless failover, moving existing connections from a failed HCA
  to the remaining HCAs in a link group
- load balancing of connections across the links in a link group

SMC-R design defines memory areas for data transport through RDMA called
RDMA Memory Buffer (RMB) which are split up into several RMB Elements
(RMBEs) assigned to single connections. This initial Linux implementation
requires real contiguous storage for these RMBs and uses a separate RMB
with just 1 RMBE for every connection. Its size is derived from:
- a system wide default minimum SMC receive buffer size (tailorable by
  /proc/net/smc/rcvbuf
- if given, the socket specific SETSOCKOPT SO_RCVBUF configuration
- the size of the tcp socket receive buffer determined from the system wide
  tcp socket receive buffer configuration
- the amount of available contiguous storage
In the Linux implementation, an equivalent concept exists for the socket
send buffers [1].

In the future, SMC-R will be enhanced to cover:
- alternate SMC-capability detection
- IPv6 support
- Tracing
- Statistic hooks

References:
[1] SMC-R Overview and Reference Materials:
    http://www-01.ibm.com/software/network/commserver/SMCR/
[2] SMC-R Informational RFC:
    https://tools.ietf.org/rfc/rfc7609
[3] Linux SMC-R Overview and Performance Summary
    (archs x86 and s390):
    http://www-01.ibm.com/software/network/commserver/SMCR/

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 net/Kconfig        |    1 +
 net/Makefile       |    1 +
 net/core/sock.c    |   20 +-
 net/smc/Kconfig    |    9 +
 net/smc/Makefile   |    3 +
 net/smc/af_smc.c   | 3136 +++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/af_smc.h   |  706 +++++++++++
 net/smc/smc_core.c | 3297 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_llc.c  | 1597 +++++++++++++++++++++++++
 net/smc/smc_llc.h  |  192 +++
 net/smc/smc_proc.c |  953 +++++++++++++++
 11 files changed, 9913 insertions(+), 2 deletions(-)
 create mode 100644 net/smc/Kconfig
 create mode 100644 net/smc/Makefile
 create mode 100644 net/smc/af_smc.c
 create mode 100644 net/smc/af_smc.h
 create mode 100644 net/smc/smc_core.c
 create mode 100644 net/smc/smc_llc.c
 create mode 100644 net/smc/smc_llc.h
 create mode 100644 net/smc/smc_proc.c

diff --git a/net/Kconfig b/net/Kconfig
index 7021c1b..57435aa 100644
--- a/net/Kconfig
+++ b/net/Kconfig
@@ -54,6 +54,7 @@ source "net/packet/Kconfig"
 source "net/unix/Kconfig"
 source "net/xfrm/Kconfig"
 source "net/iucv/Kconfig"
+source "net/smc/Kconfig"
 
 config INET
 	bool "TCP/IP networking"
diff --git a/net/Makefile b/net/Makefile
index 3995613..33a5f54 100644
--- a/net/Makefile
+++ b/net/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_MAC80211)		+= mac80211/
 obj-$(CONFIG_TIPC)		+= tipc/
 obj-$(CONFIG_NETLABEL)		+= netlabel/
 obj-$(CONFIG_IUCV)		+= iucv/
+obj-$(CONFIG_AFSMC)		+= smc/
 obj-$(CONFIG_RFKILL)		+= rfkill/
 obj-$(CONFIG_NET_9P)		+= 9p/
 obj-$(CONFIG_CAIF)		+= caif/
diff --git a/net/core/sock.c b/net/core/sock.c
index 193901d..d08ee4b 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -139,6 +139,9 @@
 
 #ifdef CONFIG_INET
 #include <net/tcp.h>
+#if IS_ENABLED(CONFIG_AFSMC)
+#include <net/smc.h>
+#endif /* CONFIG_AFSMC */
 #endif
 
 #include <net/busy_poll.h>
@@ -784,8 +787,21 @@ set_rcvbuf:
 	case SO_KEEPALIVE:
 #ifdef CONFIG_INET
 		if (sk->sk_protocol == IPPROTO_TCP &&
-		    sk->sk_type == SOCK_STREAM)
-			tcp_set_keepalive(sk, valbool);
+		    sk->sk_type == SOCK_STREAM) {
+#if IS_ENABLED(CONFIG_AFSMC)
+			if (sk->sk_family == AF_SMC) {
+				/* cannot use the TCP-way, since AFSMC
+				 * should be allowed to be a module
+				 */
+				release_sock(sk);
+				sock->ops->setsockopt(sock, SOL_TCP,
+						      SMC_KEEPALIVE,
+						      optval, optlen);
+				lock_sock(sk);
+			} else
+#endif
+				tcp_set_keepalive(sk, valbool);
+		}
 #endif
 		sock_valbool_flag(sk, SOCK_KEEPOPEN, valbool);
 		break;
diff --git a/net/smc/Kconfig b/net/smc/Kconfig
new file mode 100644
index 0000000..4ae42d5
--- /dev/null
+++ b/net/smc/Kconfig
@@ -0,0 +1,9 @@
+config AFSMC
+	tristate "AF_SMC socket address family"
+	depends on INET && INFINIBAND
+	---help---
+	  SMC_R provides a "sockets over RDMA" solution making use of
+	  RDMA over Converged Ethernet (ROCE) technology.
+	  Socket family AF_SMC implements this solution for Linux.
+
+	  Select this option if you want to run AF_SMC socket applications
diff --git a/net/smc/Makefile b/net/smc/Makefile
new file mode 100644
index 0000000..76017f0
--- /dev/null
+++ b/net/smc/Makefile
@@ -0,0 +1,3 @@
+ccflags-y += -I$(src)
+smc-y := af_smc.o smc_core.o smc_proc.o smc_llc.o
+obj-$(CONFIG_AFSMC)	+= smc.o
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
new file mode 100644
index 0000000..9f6bc19
--- /dev/null
+++ b/net/smc/af_smc.c
@@ -0,0 +1,3136 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  AF_SMC protocol family socket handler
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              Stefan Raspl <raspl@linux.vnet.ibm.com>
+ *              based on prototype from Frank Blaschka
+ */
+
+#define KMSG_COMPONENT "af_smc"
+#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/seq_file.h>
+#include <linux/uaccess.h>
+#include <linux/workqueue.h>
+#include <linux/proc_fs.h>
+#include <linux/types.h>
+#include <net/sock.h>
+#include <net/inet_sock.h>
+#include <net/tcp.h>
+#include <net/smc.h>
+#include <asm/byteorder.h>
+#include <asm/ioctls.h>
+#include <rdma/ib_verbs.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+#define VERSION "1.0"
+
+static struct proto smc_proto = {
+	.name		= "AF_SMC",
+	.owner		= THIS_MODULE,
+	.obj_size	= sizeof(struct smc_sock),
+};
+
+static struct sock *smc_sock_alloc(struct socket *, int, gfp_t);
+static void smc_tcp_listen_worker(struct work_struct *);
+static int smc_getsockopt_from_tcp(struct smc_sock *, int);
+static struct sock *smc_accept_dequeue(struct sock *, struct socket *);
+
+struct list_head smc_ib_devices;	/* list of smc_ib_devices */
+static spinlock_t smc_ib_dev_lock;	/* protect smc_ib_devices_list */
+struct smc_lgr_list_struct smc_lgr_list;	/* list of link_groups */
+struct list_head smc_lgr_freeing_list;	/* list of link groups to be freed */
+static u8 local_peerid[8] = SMC_LOCAL_PEERID_RESET;	/* unique system
+							 * identifier
+							 */
+u32 smc_ctrl_buf_cnt = 16;		/* # of control buffers per link */
+u32 smc_max_conn_per_lgr = 200;		/* max. # of connections per lgr */
+u32 smc_port_add_delay;		/* port add delay (for certain switches) */
+atomic_t smc_lgr_num = ATOMIC_INIT(0);	/* unique link group number */
+unsigned int smc_def_rcvbuf_size = 65532;	/* def. receive buffer size */
+unsigned int smc_def_sndbuf_size = 65532;	/* def. send buffer size */
+
+atomic_t smc_reconfiguring = ATOMIC_INIT(0);	/* serialize port_add/port_err
+						 * and free_lgr
+						 */
+
+#ifdef CONFIG_S390
+#include <asm/ebcdic.h>
+#endif
+
+/* link is up - establish alternate link if applicable
+ * must be called with lgr->conf_mutex taken
+ */
+void smc_link_up(struct smc_link_group *lgr)
+{
+	struct smc_link *link = NULL;
+	struct smc_roce_defs rocdefs;
+	int i;
+	u8 p1, p2;
+
+	/* determine working link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			link = &lgr->lnk[i];
+			break;
+		}
+	}
+	if (!link || (lgr->lgr_type == SYMMETRIC) ||
+	    (lgr->lgr_type == ASYMMETRIC_PEER))
+		return;
+again:
+	if (!lgr->port_up)
+		return;
+	p1 = lgr->port_up & 1;
+	p2 = lgr->port_up & 2;
+	if (lgr->role == SMC_SERV) {
+		struct llc_add_link_fake_msg add_llc;
+
+		memset(&add_llc, 0, sizeof(add_llc));
+		add_llc.hd.length = sizeof(struct llc_msg);
+		add_llc.hd.type = LLC_ADD_LINK;
+		add_llc.ibdev = lgr->ibdev_up;
+		add_llc.port_num = p1 ? p1 : p2;
+		add_llc.hd.flags |= LLC_FLAG_LOCAL;
+		llc_enqueue(link, (struct llc_msg *)&add_llc);
+		lgr->port_up &= ~add_llc.port_num;
+	} else {
+		/* invite server to start ADD LINK */
+		rocdefs.ibdev = lgr->ibdev_up;
+		rocdefs.port = p1 ? p1 : p2;
+		rocdefs.vlan = lgr->vlan;
+		smc_gid_by_dev(&rocdefs);
+		if (lgr->llc_ctl[LLC_SERV].active != LLC_GRP_NONE) {
+			mutex_unlock(&lgr->conf_mutex);
+			wait_event_interruptible_timeout(lgr->llc_waiter,
+				(lgr->llc_ctl[LLC_SERV].active == LLC_GRP_NONE),
+				LLC_WAIT_TIMEO);
+			mutex_lock(&lgr->conf_mutex);
+		}
+		if (!list_empty(&lgr->list)) {
+			if (smc_port_active(rocdefs.ibdev, rocdefs.port))
+				llc_send_add_link(link, &rocdefs, NULL, LLC_REQ,
+						  LLC_NO_NOTIFY);
+		}
+		lgr->port_up &= ~rocdefs.port;
+	}
+	goto again;
+}
+
+static void smc_port_add_worker(struct work_struct *work)
+{
+	struct smc_ib_device *smc_ibdev =
+				container_of(work, struct smc_ib_device,
+					     port_add_work.work);
+	struct smc_link_group *lgr;
+	u8 port_num = 0;
+	int i;
+
+	atomic_inc(&smc_reconfiguring);
+	for (i = 0; i < 2; i++) {
+		char pnet[SMC_MAX_PNET_ID_LEN];
+		int rc;
+
+		if (!smc_ibdev->port_add[i])
+			continue;
+		smc_check_port_attr(smc_ibdev, i);
+		if (smc_ibdev->pattr[i].state != IB_PORT_ACTIVE)
+			continue;
+		port_num = i + 1;
+		memset(pnet, 0, sizeof(pnet));
+		rc = smc_pnet_by_ibdev(port_num, pnet, smc_ibdev);
+		if (rc)
+			continue;
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+			if (strncmp(pnet, lgr->pnet_id, sizeof(pnet)) ||
+			    (lgr->lgr_type == SYMMETRIC) ||
+			    (lgr->lgr_type == ASYMMETRIC_PEER))
+				continue;
+			if (cmpxchg(&lgr->lnk_up, 0, 1) == 0) {
+				lgr->ibdev_up = smc_ibdev;
+				lgr->port_up |= port_num;
+				schedule_work(&lgr->link_up_work);
+			}
+		}
+		spin_unlock_bh(&smc_lgr_list.lock);
+		smc_ibdev->port_add[i] = 0;
+	}
+	atomic_dec(&smc_reconfiguring);
+}
+
+/* link is down - switch connections to alternate link
+ * must be called with lgr->conf_mutex taken
+ */
+void smc_link_down(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct smc_link *link = &lgr->lnk[lnk_idx];
+	struct smc_ib_device *smc_ibdev = link->roce_defs.ibdev;
+	int to_lnk_idx;
+	u8 port_num = link->roce_defs.port;
+
+	smc_modify_qp_reset(link);
+	to_lnk_idx = smc_switch_conns(lgr, lnk_idx, 1);
+	if (to_lnk_idx < 0) { /* no backup link available */
+		smc_free_link(lgr, lnk_idx);
+		return;
+	}
+	lgr->lgr_type = SINGLE;
+	if (lgr->role == SMC_SERV) {
+		struct llc_del_link_msg del_llc;
+
+		memset(&del_llc, 0, sizeof(del_llc));
+		del_llc.hd.length = sizeof(struct llc_msg);
+		del_llc.hd.type = LLC_DEL_LINK;
+		del_llc.link_num = link->link_id;
+		del_llc.reason = htonl(LLC_DEL_LOST_PATH);
+		del_llc.hd.flags |= LLC_FLAG_DEL_ORDERLY;
+		del_llc.hd.flags |= LLC_FLAG_LOCAL;
+		llc_enqueue(&lgr->lnk[to_lnk_idx], (struct llc_msg *)&del_llc);
+		/* try to restart the link */
+		if (!list_empty(&lgr->list) &&
+		    !list_empty(&smc_ibdev->list) &&
+		    smc_port_active(smc_ibdev, port_num)) {
+			lgr->ibdev_up = smc_ibdev;
+			lgr->port_up |= port_num;
+			schedule_work(&lgr->link_up_work);
+		}
+	} else {
+		if (lgr->llc_ctl[LLC_SERV].active != LLC_GRP_NONE) {
+			mutex_unlock(&lgr->conf_mutex);
+			wait_event_interruptible_timeout(lgr->llc_waiter,
+				(lgr->llc_ctl[LLC_SERV].active ==
+							LLC_GRP_NONE),
+				LLC_WAIT_TIMEO);
+			mutex_lock(&lgr->conf_mutex);
+		}
+		llc_send_del_link(&lgr->lnk[to_lnk_idx], link,
+				  LLC_FLAG_DEL_ORDERLY,
+				  LLC_DEL_LOST_PATH, LLC_NO_NOTIFY);
+	}
+}
+
+static void smc_port_err_worker(struct work_struct *work)
+{
+	struct smc_ib_device *smc_ibdev =
+		container_of(work, struct smc_ib_device, port_err_work);
+	struct smc_link_group *lgr;
+	int i, rc1, rc2;
+	u8 port_num;
+	char pnet1[SMC_MAX_PNET_ID_LEN], pnet2[SMC_MAX_PNET_ID_LEN];
+
+	atomic_inc(&smc_reconfiguring);
+	for (i = 0; i < 2; i++) {
+		if (!smc_ibdev->port_err[i])
+			continue;
+		smc_check_port_attr(smc_ibdev, i);
+		port_num = i + 1;
+		rc1 = smc_pnet_by_ibdev(1, pnet1, smc_ibdev);
+		rc2 = smc_pnet_by_ibdev(2, pnet2, smc_ibdev);
+		smc_ibdev->port_err[i] = 0;
+		if (rc1 && rc2) /* no pnet definition for this device */
+			continue;
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+			int j;
+
+			if (strncmp(pnet1, lgr->pnet_id, sizeof(pnet1)) &&
+			    strncmp(pnet2, lgr->pnet_id, sizeof(pnet2)))
+				continue;
+			if (lgr->lgr_type == NONE) /* no links */
+				continue;
+			for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+				struct smc_link *lnk = &lgr->lnk[j];
+
+				if ((lnk->roce_defs.ibdev == smc_ibdev) &&
+				    (lnk->roce_defs.port == port_num) &&
+				    cmpxchg(&lgr->lnk_down[j], 0, 1) == 0)
+					schedule_work(&lgr->link_down_work);
+			}
+		}
+		spin_unlock_bh(&smc_lgr_list.lock);
+	}
+	atomic_dec(&smc_reconfiguring);
+}
+
+static void smc_global_event_handler(struct ib_event_handler *handler,
+				     struct ib_event *event)
+{
+	struct smc_ib_device *dev;
+	u8 port_num = event->element.port_num - 1;
+
+	/* irq context! */
+	dev = container_of(handler, struct smc_ib_device, event_handler);
+	switch (event->event) {
+	case IB_EVENT_DEVICE_FATAL:
+		dev->port_err[0] = 1;
+		dev->port_err[1] = 1;
+		if ((cmpxchg(&dev->port_err[0], 0, 1) == 0) ||
+		    (cmpxchg(&dev->port_err[1], 0, 1) == 0))
+			schedule_work(&dev->port_err_work);
+		break;
+	case IB_EVENT_PORT_ERR:
+		if (cmpxchg(&dev->port_err[port_num], 0, 1) == 0)
+			schedule_work(&dev->port_err_work);
+		break;
+	case IB_EVENT_PORT_ACTIVE:
+		if (cmpxchg(&dev->port_add[port_num], 0, 1) == 0)
+			schedule_delayed_work(&dev->port_add_work,
+					      smc_port_add_delay);
+		break;
+	case IB_EVENT_GID_CHANGE:
+		break;
+	default:
+		break;
+	}
+}
+
+static void smc_ib_add_dev(struct ib_device *dev)
+{
+	struct smc_ib_device *smc_ibdev;
+	struct smc_roce_defs rocdefs;
+	struct ib_cq_init_attr attr = {.cqe = SMC_MAX_CQE, .comp_vector = 0};
+
+	if (dev->node_type != RDMA_NODE_IB_CA)
+		return;
+	smc_ibdev = kzalloc(sizeof(*smc_ibdev), GFP_KERNEL);
+	if (!smc_ibdev)
+		return;
+	smc_ibdev->dev = dev;
+	smc_ibdev->roce_cq_send = ib_create_cq(dev, smc_cq_handler_send,
+		smc_cq_event_handler, smc_ibdev, &attr);
+	if (IS_ERR(smc_ibdev->roce_cq_send))
+		goto err;
+	smc_ibdev->roce_cq_recv = ib_create_cq(dev, smc_cq_handler_recv,
+		smc_cq_event_handler, smc_ibdev, &attr);
+	if (IS_ERR(smc_ibdev->roce_cq_recv))
+		goto err;
+	if (!strncmp(local_peerid, SMC_LOCAL_PEERID_RESET,
+		     sizeof(local_peerid))) {
+		rocdefs.ibdev = smc_ibdev;
+		rocdefs.port = 1;
+		smc_get_ib_mac(&rocdefs);
+		memcpy(&local_peerid[2], &rocdefs.mac, sizeof(rocdefs.mac));
+		get_random_bytes_arch(&local_peerid[0], 2);
+	}
+	spin_lock_bh(&smc_ib_dev_lock);
+	list_add_tail(&smc_ibdev->list, &smc_ib_devices);
+	spin_unlock_bh(&smc_ib_dev_lock);
+	ib_set_client_data(dev, &smc_ib_client, smc_ibdev);
+	smc_check_dev_attr(smc_ibdev);
+	INIT_WORK(&smc_ibdev->port_err_work, smc_port_err_worker);
+	INIT_DELAYED_WORK(&smc_ibdev->port_add_work, smc_port_add_worker);
+	INIT_IB_EVENT_HANDLER(&smc_ibdev->event_handler, dev,
+			      smc_global_event_handler);
+	ib_register_event_handler(&smc_ibdev->event_handler);
+	tasklet_init(&smc_ibdev->recv_tasklet, smc_recv_tasklet_fn,
+		     (unsigned long)smc_ibdev);
+	tasklet_init(&smc_ibdev->send_tasklet, smc_send_tasklet_fn,
+		     (unsigned long)smc_ibdev);
+	return;
+
+err:
+	kfree(smc_ibdev);
+}
+
+static void smc_wait_for_freed_links(struct smc_ib_device *sdev)
+{
+	struct smc_link_group *lgr;
+	struct smc_link *lnk;
+	int i;
+
+again:
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			lnk = &lgr->lnk[i];
+			if (lnk->roce_defs.ibdev == sdev) {
+				spin_unlock_bh(&smc_lgr_list.lock);
+				switch (atomic_read(&lnk->state)) {
+				case SMC_LINK_FREED:
+				case SMC_LINK_UP: /* wait for port_err finish */
+					msleep(LLC_WAIT_FIRST_TIMEO);
+					break;
+				case SMC_LINK_DOWN:
+					smc_free_link(lgr, i);
+					break;
+				default:
+					break;
+				}
+				goto again;
+			}
+		}
+	}
+	list_for_each_entry(lgr, &smc_lgr_freeing_list, list) {
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			lnk = &lgr->lnk[i];
+			if (lnk->roce_defs.ibdev == sdev) {
+				spin_unlock_bh(&smc_lgr_list.lock);
+				switch (atomic_read(&lnk->state)) {
+				case SMC_LINK_FREED:
+					msleep(LLC_WAIT_FIRST_TIMEO);
+					break;
+				case SMC_LINK_UP:
+					atomic_set(&lnk->state, SMC_LINK_DOWN);
+					/* fall through */
+				case SMC_LINK_DOWN:
+					smc_free_link(lgr, i);
+					break;
+				default:
+					break;
+				}
+				goto again;
+			}
+		}
+	}
+	spin_unlock_bh(&smc_lgr_list.lock);
+}
+
+static void smc_ib_remove_dev(struct ib_device *dev)
+{
+	struct smc_ib_device *smc_ibdev;
+
+	atomic_inc(&smc_reconfiguring);
+	smc_ibdev = ib_get_client_data(dev, &smc_ib_client);
+	ib_unregister_event_handler(&smc_ibdev->event_handler);
+	smc_ibdev->port_add[0] = 0;
+	smc_ibdev->port_add[1] = 0;
+	smc_ibdev->port_err[0] = 1;
+	smc_ibdev->port_err[1] = 1;
+	schedule_work(&smc_ibdev->port_err_work);
+	ib_set_client_data(dev, &smc_ib_client, NULL);
+	spin_lock_bh(&smc_ib_dev_lock);
+	list_del_init(&smc_ibdev->list);
+	spin_unlock_bh(&smc_ib_dev_lock);
+	smc_wait_for_freed_links(smc_ibdev);
+	tasklet_kill(&smc_ibdev->recv_tasklet);
+	tasklet_kill(&smc_ibdev->send_tasklet);
+	ib_destroy_cq(smc_ibdev->roce_cq_send);
+	ib_destroy_cq(smc_ibdev->roce_cq_recv);
+	flush_work(&smc_ibdev->port_err_work);
+	kfree(smc_ibdev);
+	atomic_dec(&smc_reconfiguring);
+}
+
+struct ib_client smc_ib_client = {
+	.name	= "smc_ib",
+	.add	= smc_ib_add_dev,
+	.remove = smc_ib_remove_dev
+};
+
+void smc_sock_wake_rx(struct sock *sk)
+{
+	struct socket_wq *wq;
+
+	/* derived from sock_def_readable() */
+	/* Attention: sk_data_ready=smc_sock_wake_rx called in listen_worker */
+	rcu_read_lock();
+	wq = rcu_dereference(sk->sk_wq);
+	if (wq_has_sleeper(wq)) {
+		wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLPRI |
+						POLLRDNORM | POLLRDBAND);
+	}
+	if ((sk->sk_shutdown == SHUTDOWN_MASK) ||
+	    (sk->sk_state == SMC_CLOSED))
+		sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_HUP);
+	else
+		sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
+	rcu_read_unlock();
+}
+
+void smc_sock_wake_tx(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct socket *sock = sk->sk_socket;
+	struct socket_wq *wq;
+
+/* similar to sk_stream_write_space */
+	if (atomic_read(&smc->conn.tx_buf_space) && sock) {
+		clear_bit(SOCK_NOSPACE, &sock->flags);
+		rcu_read_lock();
+		wq = rcu_dereference(sk->sk_wq);
+		if (wq_has_sleeper(wq)) {
+			wake_up_interruptible_poll(&wq->wait,
+						   POLLOUT | POLLWRNORM |
+						   POLLWRBAND);
+		}
+		if (wq && wq->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN))
+			sock_wake_async(sock, SOCK_WAKE_SPACE, POLL_OUT);
+		rcu_read_unlock();
+	}
+}
+
+static void smc_sk_release(struct smc_sock *, int);
+static void smc_sock_destruct(struct sock *sk);
+
+static void smc_destruct_non_accepted(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int i;
+
+	sock_hold(sk);
+	lock_sock(sk);
+	smc_sk_release(smc, 0);
+	if (smc->tcpsocket) {
+		struct socket *tcp;
+
+		tcp = smc->tcpsocket;
+		smc->tcpsocket = NULL;
+		sock_release(tcp);
+	}
+	release_sock(sk);
+	if (!smc->use_tcp) {
+		for (i = 0;
+		     ((i < 10) &&
+		      (smc_pending_sends(conn) ||
+		       !smc_close_received(conn)));
+		     i++) {
+			msleep(SMC_WAIT_PENDING_SENDS_TIMEO);
+		}
+	}
+	if (cmpxchg(&smc->sock_put_done, 0, 1) == 0) {
+		sock_set_flag(sk, SOCK_ZAPPED);
+		sock_set_flag(sk, SOCK_DEAD);
+		schedule_work(&smc->release_work);
+	}
+	sock_put(sk);
+}
+
+static void smc_sock_cleanup_listen(struct sock *parent)
+{
+	struct sock *sk;
+
+	/* Close non-accepted connections */
+	while ((sk = smc_accept_dequeue(parent, NULL)))
+		smc_destruct_non_accepted(sk);
+}
+
+static void smc_wait_close(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	signed long timeout;
+	DEFINE_WAIT(wait);
+
+	/* cancel CORK */
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+		schedule_delayed_work(&smc->write_work, 0);
+	}
+	timeout = sk->sk_lingertime ? sk->sk_lingertime : MAX_SCHEDULE_TIMEOUT;
+	if (smc_prepared_sends(&smc->conn) && !sk->sk_err &&
+	    !(current->flags & PF_EXITING)) {
+		release_sock(sk);
+		wait_event_interruptible_timeout(smc->prep_waiter,
+						!smc_prepared_sends(&smc->conn),
+						timeout);
+		lock_sock(sk);
+	}
+	timeout = SMC_WAIT_PENDING_SENDS_TIMEO;
+	if (smc_pending_sends(&smc->conn) &&
+	    !(current->flags & PF_EXITING)) {
+		set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+		do {
+			prepare_to_wait(sk_sleep(sk), &wait,
+					TASK_INTERRUPTIBLE);
+			if (sk_wait_event(sk, &timeout,
+					  !smc_pending_sends(&smc->conn)))
+				break;
+		} while (!signal_pending(current) && timeout);
+		clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+		finish_wait(sk_sleep(sk), &wait);
+	}
+}
+
+static int smc_wait_for_pending_sends(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	u32 i, j;
+	int rc;
+
+	release_sock(&smc->sk);
+	rc = wait_event_timeout(smc->destruct_waiter,
+				!smc_pending_sends(&smc->conn),
+				SMC_WAIT_PENDING_SENDS_TIMEO);
+	lock_sock(&smc->sk);
+	if (!rc) { /* timeout reached */
+		for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+			link = &smc->conn.lgr->lnk[j];
+			if (!link->link_id)
+				continue;
+			for (i = 0; i < link->send_wr_num; i++)
+				if (link->pending[i].conn == &smc->conn) {
+					link->pending[i].conn = NULL;
+					clear_bit(i, link->pending_bit_mask);
+				}
+		}
+	}
+	return rc;
+}
+
+static void smc_conn_term_abnormal(struct smc_sock *smc, int old_state)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+
+	sk_stop_timer(sk, &sk->sk_timer);
+	while (work_busy(&smc->timer_work))
+		msleep(20);
+	if (sk->sk_state == SMC_PEERCLW2 ||
+	    sk->sk_state == SMC_PEERFINCLW ||
+	    sk->sk_state == SMC_CLOSED ||
+	    sk->sk_state == SMC_INIT ||
+	    sk->sk_state == SMC_PEERABORTW) {
+		sk->sk_state = SMC_CLOSED;
+		if (timer_pending(&smc->fin_timer))
+			mod_timer_pending(&smc->fin_timer, jiffies);
+	} else {
+		if (sk->sk_state == SMC_PEERCLW1) {
+			/* enable wait_close wakeup */
+			smc_clear_pending_sends(conn);
+		}
+		/* cancel close waiting */
+		atomic64_set(&conn->tx_curs_prep.s.acurs,
+			     conn->tx_curs_sent.s.lcurs);
+		atomic64_set(&conn->tx_curs_fin.s.acurs,
+			     conn->tx_curs_sent.s.lcurs);
+		sk->sk_state = SMC_PROCESSABORT;
+		/* purge all in-flight data, do abnormal term step */
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+		if ((old_state != SMC_PROCESSABORT) && (conn->lgr) &&
+		    (conn->lgr->lgr_type != NONE)) {
+			smc_ctrl_send(conn, conn->lnk_idx);
+			if (conn->lgr->lgr_type != SINGLE)
+				smc_send_close_alt(smc);
+		}
+		if (sock_flag(sk, SOCK_DEAD))
+			mod_timer_pending(&smc->fin_timer,
+					  jiffies + TCP_TIMEWAIT_LEN);
+	}
+	sk->sk_err = ECONNABORTED;
+}
+
+static void smc_accept_unlink(struct sock *);
+
+void smc_conn_term(struct smc_sock *smc)
+{
+	struct sock *tcpsk;
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int old_state;
+
+	old_state = sk->sk_state;
+	/* Handle abnormal termination */
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		smc_conn_term_abnormal(smc, old_state);
+		goto out;
+	}
+	switch (sk->sk_state) {
+	/* Normal termination - Passive close part */
+	case SMC_INIT:
+	case SMC_ACTIVE:
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done ||
+		    conn->local_rx_ctrl.conn_state_flags.closed_conn) {
+			/* complete any outstanding recv with zero-length
+			 * if peerclosedconn and pending data to be written
+			 * then reset conn
+			 */
+			sk->sk_state = SMC_APPLCLW1;
+			if (sock_flag(sk, SOCK_DEAD))
+				mod_timer(&smc->fin_timer,
+					  jiffies + TCP_TIMEWAIT_LEN);
+		}
+		break;
+	case SMC_PEERFINCLW:
+		if (conn->local_rx_ctrl.conn_state_flags.closed_conn)
+			sk->sk_state = SMC_CLOSED;
+		break;
+		/* Normal termination - Active close part */
+	case SMC_PEERCLW1:
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done) {
+			/* complete any outstanding recv with zero-length */
+			sk->sk_state = SMC_PEERCLW2;
+		} /* fall through */
+	case SMC_PEERCLW2:
+		if (conn->local_rx_ctrl.conn_state_flags.closed_conn) {
+			struct smc_e_ctrl *tx_ctrl = &conn->local_tx_ctrl;
+
+			/* complete any outstanding recv with zero-length */
+			if (sk->sk_shutdown == SHUTDOWN_MASK &&
+			    (tx_ctrl->conn_state_flags.closed_conn ||
+			     tx_ctrl->conn_state_flags.abnormal_close)) {
+				sk->sk_state = SMC_CLOSED;
+				if (timer_pending(&smc->fin_timer))
+					mod_timer_pending(&smc->fin_timer,
+							  jiffies);
+			} else {
+				sk->sk_state = SMC_APPLFINCLW;
+			}
+		}
+		break;
+	default:
+		break;
+	}
+out:
+	if ((sk->sk_err == ECONNABORTED) &&
+	    !list_empty(&smc->accept_q))
+		smc_accept_unlink(sk);
+	if (smc_stop_received(conn)) {
+		sk->sk_shutdown = sk->sk_shutdown | RCV_SHUTDOWN;
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			tcpsk = smc->tcpsocket->sk;
+			tcpsk->sk_shutdown = tcpsk->sk_shutdown | RCV_SHUTDOWN;
+		}
+	}
+	if (smc_close_received(conn) &&
+	    (sk->sk_state == SMC_CLOSED) &&
+	    !sk->sk_socket &&
+	    !smc_pending_sends(conn))
+		sock_set_flag(sk, SOCK_DEAD);
+	if ((old_state != sk->sk_state) &&
+	    (old_state != SMC_INIT))
+		sk->sk_state_change(sk);
+}
+
+static void smc_sk_release(struct smc_sock *smc, int wait)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+
+	if (smc->use_tcp) {
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		goto out;
+	}
+	sk_stop_timer(sk, &sk->sk_timer);
+	cancel_work_sync(&smc->timer_work);
+
+	sk->sk_shutdown = SHUTDOWN_MASK;
+
+	switch (sk->sk_state) {
+	case SMC_INIT:
+	case SMC_CLOSED:
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		break;
+	case SMC_LISTEN:
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			rc = kernel_sock_shutdown(smc->tcpsocket, SHUT_RDWR);
+			smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+			/* to wake up kernel_accept of smc_tcp_listen_worker */
+		}
+		release_sock(sk);
+		smc_sock_cleanup_listen(sk);
+		while (work_busy(&smc->tcp_listen_work) ||
+		       work_busy(&smc->listen_work)) {
+			msleep(20);
+		}
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+		break;
+	case SMC_ACTIVE:
+		/* active close */
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done)
+			sk->sk_state = SMC_PEERCLW2;
+		else
+			sk->sk_state = SMC_PEERCLW1;
+		sk->sk_state_change(sk);
+		/* handle SOCK_LINGER */
+		/* fall through */
+	case SMC_PEERCLW1:
+		if (wait)
+			smc_wait_close(smc);
+		sk->sk_state = SMC_PEERCLW2;
+		smc_send_close(smc);
+		/* fall through */
+	case SMC_PEERCLW2:
+		rc = smc_wait_for_pending_sends(smc);
+		if ((rc <= 0) ||   /* timeout or interrupted */
+		    smc_close_received(conn)) {
+			sk->sk_state = SMC_CLOSED;
+			sk->sk_state_change(sk);
+		}
+		break;
+	case SMC_PROCESSABORT:
+	case SMC_APPLFINCLW:
+		/* socket already shutdown wr or both (active close) */
+		if (!conn->local_tx_ctrl.conn_state_flags.closed_conn &&
+		    conn->lgr) {
+			rc = smc_send_close(smc);
+			smc_wait_for_pending_sends(smc);
+		}
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		break;
+	case SMC_APPLCLW1:
+		/* passive close */
+		if (wait)
+			smc_wait_close(smc);
+		/* fall through */
+	case SMC_APPLCLW2:
+		/* confirm close from peer */
+		smc_send_close(smc);
+		if (smc_close_received(conn)) {
+			smc_wait_for_pending_sends(smc);
+			sk->sk_state = SMC_CLOSED;
+			sk->sk_state_change(sk);
+		} else {
+			sk->sk_state = SMC_PEERFINCLW;
+			sk->sk_state_change(sk);
+		}
+		break;
+	case SMC_PEERFINCLW:
+		break;
+	default:
+		break;
+	}
+out:
+	return;
+}
+
+/* determine vitality of peer
+ * returns 0 if peer is alive
+ */
+int smc_check_peer_gone(struct smc_link_group *lgr)
+{
+	struct smc_link *link;
+	struct llc_qentry *qentry = NULL;
+	int i, rc = 0;
+
+	if (!lgr)
+		return -EPIPE;
+	if (signal_pending(current))
+		return 0;
+	/* check peer with testlink */
+	if (!llc_initiate(lgr, LLC_GRP_TEST_LINK))
+		return -ETIMEDOUT;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		link = &lgr->lnk[i];
+		if (atomic_read(&link->state) != SMC_LINK_UP)
+			continue;
+		rc = llc_send_test_link(link, "KEEPALIVE_CHECK");
+		if (rc)
+			continue;
+		qentry = llc_wait(lgr, LLC_TESTLINK, LLC_WAIT_TIMER_TIMEO,
+				  LLC_TEST_LINK);
+		if (qentry) { /* peer is alive */
+			kfree(qentry);
+			goto out;
+		}
+	}
+	rc = -ETIMEDOUT;
+out:
+	llc_stop(lgr, LLC_GRP_TEST_LINK);
+	return rc;
+}
+
+static int smc_sock_release(struct socket *sock)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc;
+	int rc = 0;
+
+	if (!sk || (sk->sk_state == SMC_DESTRUCT))
+		return rc;
+	smc = smc_sk(sk);
+
+	sock_hold(sk);
+	if (sk->sk_state == SMC_LISTEN)
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+	else
+		lock_sock(sk);
+	smc_sk_release(smc, 1);
+	if (smc->tcpsocket) {
+		struct socket *tcp;
+
+		tcp = smc->tcpsocket;
+		smc->tcpsocket = NULL;
+		sock_release(tcp);
+	}
+
+	/* detach socket */
+	sock->state = SS_UNCONNECTED;
+	sock_set_flag(sk, SOCK_ZAPPED);
+	sock_orphan(sk); /* incl. SOCK_DEAD */
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+	}
+	release_sock(sk);
+	if (!smc->use_tcp && (smc->write_work.work.func == smc_write_worker)) {
+		wake_up_interruptible(&smc->rx_waiter);
+		cancel_delayed_work_sync(&smc->write_work);
+	}
+	if (sk->sk_state == SMC_PEERCLW2) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			/* guarantee socket cleanup */
+			mod_timer_pending(&smc->fin_timer,
+					  jiffies + TCP_FIN_TIMEOUT);
+			smc_terminate_conn(smc->conn.lgr);
+		}
+	}
+
+	/* check if we can cleanup sock or keep it for peers responses */
+	if ((sk->sk_state == SMC_CLOSED) &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		if (timer_pending(&smc->fin_timer))
+			mod_timer_pending(&smc->fin_timer, jiffies);
+		schedule_work(&smc->release_work);
+	}
+	sock->sk = NULL;
+	sock_put(sk);
+
+	return rc;
+}
+
+static int smc_sock_bind(struct socket *sock, struct sockaddr *addr,
+			 int addr_len)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc;
+
+	smc->tcpsocket->sk->sk_reuse = sk->sk_reuse;
+	if (addr->sa_family == AF_SMC)
+		addr->sa_family = AF_INET;
+	rc = kernel_bind(smc->tcpsocket, addr, addr_len);
+	return rc;
+}
+
+/* Wait for data on the tcp-socket and fallback to tcp if decline is received
+ * Returns 0 if success
+ *         tcpsocket error, -ETIMEDOUT, -EPROTO or -ECONNABORTED otherwise
+ */
+int clc_wait_msg(struct smc_sock *smc, int noblock, char *buf, int buflen,
+		 u8 type, int *reason_code)
+{
+	struct sock *sk = smc->tcpsocket->sk;
+	long timeo;
+	int rc = 0;
+	int len;
+	int krflags = 0;
+	struct msghdr msg;
+	struct kvec vec;
+	struct smc_clc_msg_hdr *clcm = (struct smc_clc_msg_hdr *)buf;
+	DEFINE_WAIT(wait);
+
+	timeo = (noblock) ? 0 : LLC_WAIT_FIRST_TIMEO + 1 * HZ;
+	if (noblock)
+		krflags = MSG_DONTWAIT;
+
+again:
+	lock_sock(sk);
+	rc = sk_wait_data(sk, &timeo, NULL);
+	release_sock(sk);
+	if (sk->sk_err)
+		rc = -sk->sk_err;
+	if (signal_pending(current)) {
+		rc = -EINTR;
+		sk->sk_err = -rc;
+	}
+	if (rc < 0) {
+		smc->sk.sk_err = sk->sk_err;
+		goto out;
+	}
+	if ((rc == 0) && !noblock) {
+		if (timeo) {
+			goto again;
+		} else {
+			smc->sk.sk_err = ETIMEDOUT;
+			rc = -ETIMEDOUT;
+			goto out;
+		}
+	}
+	rc = 0;
+	vec.iov_base = buf;
+	vec.iov_len = buflen;
+	memset(&msg, 0, sizeof(struct msghdr));
+	krflags |=  MSG_PEEK; /* peek the first few bytes */
+	len = kernel_recvmsg(smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_decline_clc_msg), krflags);
+	if (!len) { /* no data */
+		smc->sk.sk_err = ETIMEDOUT;
+		rc = -ETIMEDOUT;
+		goto out;
+	}
+	if ((len < sizeof(struct smc_decline_clc_msg)) ||
+	    memcmp(clcm->ec_clce, SMC_EC, sizeof(SMC_EC)) ||
+	    ((clcm->type != SMC_CLC_DECLINE) && (clcm->type != type))) {
+		smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	len = ntohs(clcm->length);
+	if (!len) {
+		smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	vec.iov_base = buf;
+	vec.iov_len = buflen;
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_recvmsg(smc->tcpsocket, &msg, &vec, 1,
+			     len /* - sizeof(struct smc_clc_msg_hdr) */, 0);
+	if (clcm->type == SMC_CLC_DECLINE) {
+		smc->use_tcp = 1;
+		if (clcm->flags & SMC_LINK_GROUP_OUT_OF_SYNCH) {
+			smc_terminate_conn(smc->conn.lgr);
+			sk->sk_err = ECONNABORTED;
+			rc = -ECONNABORTED;
+			return rc;
+		}
+		*reason_code = SMC_CLC_DEC_REPLY;
+		goto out;
+	}
+out:
+	return rc;
+}
+
+int clc_send_decline(struct smc_sock *smc, u32 reason_code, u8 flags)
+{
+	struct smc_decline_clc_msg dclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int len;
+
+	memset(&dclc, 0, sizeof(struct smc_decline_clc_msg));
+	memcpy(dclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(dclc.peer_id, local_peerid, sizeof(local_peerid));
+	dclc.hdr.type = SMC_CLC_DECLINE;
+	dclc.hdr.length = htons(sizeof(struct smc_decline_clc_msg));
+	dclc.hdr.flags = SMC_CLC_V1 | flags;
+	dclc.peer_diagnosis = htonl(reason_code);
+	memcpy(dclc.ec_dclcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &dclc;
+	vec.iov_len = sizeof(struct smc_decline_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_decline_clc_msg));
+	if (len < sizeof(struct smc_decline_clc_msg))
+		smc->sk.sk_err = EPROTO;
+	if (len < 0)
+		smc->sk.sk_err = -len;
+	return len;
+}
+
+/* Copy flags relevant for SO_* sockopts */
+static void smc_copy_sockopt_settings(struct sock *nsk, struct sock *osk)
+{
+	nsk->sk_type = osk->sk_type;
+	nsk->sk_sndbuf = osk->sk_sndbuf;
+	nsk->sk_rcvbuf = osk->sk_rcvbuf;
+	nsk->sk_sndtimeo = osk->sk_sndtimeo;
+	nsk->sk_rcvtimeo = osk->sk_rcvtimeo;
+	nsk->sk_mark = osk->sk_mark;
+	nsk->sk_priority = osk->sk_priority;
+	nsk->sk_rcvlowat = osk->sk_rcvlowat;
+	nsk->sk_bound_dev_if = osk->sk_bound_dev_if;
+	sock_copy_flags(nsk, osk);
+}
+
+static void smc_init_tmp_sockopts(struct smc_sock *smc)
+{
+	smc->tmp_sockopts.set = 0;
+	smc->tmp_sockopts.cork = 0;
+	smc->tmp_sockopts.defer_accept = 0;
+	smc->tmp_sockopts.nodelay = 0;
+}
+
+static int smc_sock_setsockopt(struct socket *, int, int, char __user *,
+			       unsigned int);
+
+/* Apply all user-set sockopts effective that were supressed as they
+ * might have a negative effect during the CLC handshake
+ */
+static int smc_apply_tmp_sockopts(struct socket *socket,
+				  const struct smc_tmp_sockopts *sopts)
+{
+	int rc = 0, brc;
+	mm_segment_t old_fs = get_fs();
+
+	set_fs(KERNEL_DS);
+	if (sopts->set & SMC_SOCKOPTS_CORK) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_CORK,
+					  (char __user *)&sopts->cork,
+					  sizeof(sopts->cork));
+		if (brc)
+			rc = -EFAULT;
+	}
+	if (sopts->set & SMC_SOCKOPTS_DEFER_ACCEPT) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_DEFER_ACCEPT,
+					  (char __user *)&sopts->defer_accept,
+					  sizeof(sopts->defer_accept));
+		if (brc)
+			rc = -EFAULT;
+	}
+	if (sopts->set & SMC_SOCKOPTS_NODELAY) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_NODELAY,
+					  (char __user *)&sopts->nodelay,
+					  sizeof(sopts->nodelay));
+		if (brc)
+			rc = -EFAULT;
+	} else {
+		struct smc_sock *smc = smc_sk(socket->sk);
+
+		tcp_sk(smc->tcpsocket->sk)->nonagle &= ~TCP_NAGLE_OFF;
+	}
+	set_fs(old_fs);
+
+	return rc;
+}
+
+static int smc_find_link_accept(struct smc_link_group *lgr,
+				struct smc_acc_conf_clc_msg *clc)
+{
+	int i;
+	u32 tmp_qpnum;
+
+	ntoh_three(&tmp_qpnum, clc->qp_num);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (!memcmp(&lgr->lnk[i].gid_peer, &clc->hdr2.gid, 16) &&
+		    !memcmp(&lgr->lnk[i].mac_peer, &clc->hdr2.mac, 6) &&
+		    (lgr->lnk[i].qp_num_peer == tmp_qpnum))
+			return i;
+	}
+	return -ENOENT;
+}
+
+static int smc_rtoken_handling(struct smc_link_group *lgr, struct smc_sock *smc,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	struct smc_connection *conn = &smc->conn;
+	u64 vaddr = be64_to_cpu(clc->rmb_vaddr);
+	unsigned int rkey = ntohl(clc->rmb_rkey);
+	int lnk_idx;
+
+	lnk_idx = smc_find_link_accept(lgr, clc);
+	if (lnk_idx < 0)
+		return -ENOENT;
+	conn->rtok_idx =
+		smc_find_rtoken_by_link(lgr, lnk_idx, rkey);
+	if (conn->rtok_idx == -ENOENT) {
+		conn->rtok_idx = smc_get_rtoken(lgr);
+		lgr->rtok[conn->rtok_idx].rkey[lnk_idx] = rkey;
+		lgr->rtok[conn->rtok_idx].vaddr[lnk_idx] = vaddr;
+		lgr->rtok[conn->rtok_idx].link_id[lnk_idx] =
+			lgr->lnk[lnk_idx].link_id;
+	}
+	return 0;
+}
+
+static int smc_send_proposal(struct smc_sock *smc,
+			     struct smc_roce_defs *roce_sugg,
+			     struct smc_acc_conf_clc_msg *aclc,
+			     int *reason_code)
+{
+	int pclc_msgsize = sizeof(struct smc_proposal_clc_msg) +
+			   sizeof(struct smc_proposal_clc_msg2);
+	struct smc_proposal_clc_msg *pclc;
+	struct smc_proposal_clc_msg2 *pclc2;
+	struct kvec vec;
+	struct msghdr msg;
+	int rc = 0;
+
+	pclc = kzalloc(pclc_msgsize, GFP_KERNEL);
+	if (!pclc)
+		return -ENOMEM;
+	/* send SMC Proposal CLC message (peer ID, GID, MAC, VLAN) */
+	memcpy(&pclc->hdr2.gid, &roce_sugg->gid, sizeof(roce_sugg->gid));
+	memcpy(&pclc->hdr2.mac, &roce_sugg->mac, sizeof(roce_sugg->mac));
+	pclc->iparea_offset = 0;
+
+	pclc2 = (struct smc_proposal_clc_msg2 *)
+		((char *)pclc + sizeof(struct smc_proposal_clc_msg) +
+		 pclc->iparea_offset);
+
+	/* is it sufficient to use IP from connection itself? */
+	rc = smc_netinfo_by_tcpsk(&pclc2->outgoing_subnet,
+				  &pclc2->subnet_mask_bits, smc->tcpsocket);
+	memcpy(&pclc->hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(&pclc->hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	pclc->hdr.type = SMC_CLC_PROPOSAL;
+	pclc->hdr.length = htons(pclc_msgsize);
+	pclc->hdr.flags = SMC_CLC_V1;
+	memcpy(&pclc2->ec_pclcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = pclc;
+	vec.iov_len = pclc_msgsize;
+	memset(&msg, 0, sizeof(struct msghdr));
+	rc = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1, pclc_msgsize);
+	kfree(pclc);
+	if (rc >= 0) {
+		if (rc < pclc_msgsize)
+			rc = -EPROTO;
+		else
+			rc = 0;
+	}
+	if (rc) {
+		smc->sk.sk_err = -rc;
+		return rc;
+	}
+
+	/* receive SMC Accept CLC message (GID, MAC, VLAN, QP )*/
+	rc = clc_wait_msg(smc, SMC_CLC_BLOCK, (char *)aclc,
+			  sizeof(struct smc_acc_conf_clc_msg),
+			  SMC_CLC_ACCEPTANCE, reason_code);
+	return rc;
+}
+
+static int smc_send_confirm(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct smc_connection *conn = &smc->conn;
+	struct smc_acc_conf_clc_msg cclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int rc = 0;
+
+	link = &conn->lgr->lnk[conn->lnk_idx];
+	lgr = link->lnk_grp;
+	/* send SMC Confirm CLC msg (MAC, GID, QP, link num, link user id) */
+	memset(&cclc, 0, sizeof(struct smc_acc_conf_clc_msg));
+	cclc.tcp_conn_idx = 1; /* as long as 1 RMB = 1 RMBE */
+	memcpy(cclc.hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	memcpy(&cclc.hdr2.gid, &link->roce_defs.gid,
+	       sizeof(link->roce_defs.gid));
+	memcpy(&cclc.hdr2.mac, &link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+	hton_three(link->roce_qp->qp_num, cclc.qp_num);
+	hton_three(link->psn_initial, cclc.psn);
+	cclc.rmb_rkey =
+		htonl(conn->rmb_rx_elem->mr_rx[conn->lnk_idx]->rkey);
+	cclc.rmb_vaddr =
+		cpu_to_be64((u64)conn->rmb_rx_elem->rmb_rx_dma[conn->lnk_idx]);
+	memcpy(cclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	cclc.hdr.type = SMC_CLC_CONFIRMATION;
+	cclc.hdr.length = htons(sizeof(struct smc_acc_conf_clc_msg));
+	cclc.hdr.flags = SMC_CLC_V1;
+	cclc.rmb_alert_token = conn->alert_token_local;
+	cclc.flags2 = conn->rmb_rx_size_short +
+		      min(link->path_mtu, link->mtu_peer);
+	memcpy(cclc.ec_clcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &cclc;
+	vec.iov_len = sizeof(struct smc_acc_conf_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	rc = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1,
+			    sizeof(struct smc_acc_conf_clc_msg));
+	if (rc >= 0) {
+		if (rc < sizeof(struct smc_acc_conf_clc_msg))
+			rc = -EPROTO;
+		else
+			rc = 0;
+	}
+	if (rc)
+		smc->sk.sk_err = -rc;
+	return rc;
+}
+
+static int smc_initial_map_rmb(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &lgr->lnk[smc->conn.lnk_idx];
+	long rc = 0;
+
+	if (!smc->conn.tx_rmb_reused) {
+		rc = smc_map_rmbs_tx(lgr, smc->conn.lnk_idx,
+				     smc->conn.rmb_tx_size,
+				     smc->conn.rmb_tx_elem);
+		if (rc)
+			goto out;
+	}
+	if (!smc->conn.rx_rmb_reused) {
+		rc = smc_map_rmbs_rx(lgr, smc->conn.lnk_idx,
+				     smc->conn.rmb_rx_size,
+				     smc->conn.rmb_rx_elem);
+		if (rc)
+			goto out;
+		rc = smc_get_dma_mr(lgr, smc->conn.lnk_idx,
+				    smc->conn.rmb_rx_elem);
+		if (rc)
+			goto out;
+	}
+	return 0;
+
+out:
+	if (rc && smc_lnk_downing(&link->state)) {
+		release_sock(&smc->sk);
+		smc_link_down(lgr, link - &lgr->lnk[0]);
+		lock_sock(&smc->sk);
+	}
+	return (int)rc;
+}
+
+static int smc_announce_rmb(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* Announce new RMB */
+	if (smc->conn.tx_rmb_reused && smc->conn.rx_rmb_reused)
+		goto out;
+	if (!llc_initiate(lgr, LLC_GRP_CONF_RKEY)) { /* timeout */
+		*reason_code = SMC_CLC_DEC_TU;
+		rc = -ETIMEDOUT;
+		goto out;
+	}
+	smc_map_rmbs(smc);
+	if (lgr->lgr_type == SINGLE)
+		goto out_stop;
+	if (smc->conn.rx_rmb_reused)
+		goto out_stop;
+	rc = llc_do_confirm_rkey(link, smc->conn.rmb_rx_elem);
+	if (rc >= 0) {
+		if (lgr->role == SMC_CLNT)
+			release_sock(&smc->sk);
+		qentry = llc_wait(lgr, lgr->role, LLC_WAIT_TIMEO,
+				  LLC_CONFIRM_RKEY);
+		if (lgr->role == SMC_CLNT)
+			lock_sock(&smc->sk);
+		if ((!qentry) ||
+		    (qentry->msg.hd.flags & LLC_FLAG_RKEY_NEG)) {
+			kfree(qentry);
+			if (smc_lnk_downing(&link->state)) {
+				if (lgr->role == SMC_CLNT)
+					release_sock(&smc->sk);
+				smc_link_down(lgr, link - &lgr->lnk[0]);
+				if (lgr->role == SMC_CLNT)
+					lock_sock(&smc->sk);
+			}
+			*reason_code = SMC_CLC_DEC_TU;
+			rc = -ETIMEDOUT;
+			goto out_stop;
+		}
+		kfree(qentry);
+		rc = 0;
+	} else {
+		*reason_code = SMC_CLC_DEC_SEND;
+		rc = -EPIPE;
+	}
+out_stop:
+	llc_stop(lgr, LLC_GRP_CONF_RKEY);
+out:
+	if (rc)
+		smc_remove_rx_rmb(smc);
+	return rc;
+}
+
+static int smc_clnt_conf_first_lnk(struct smc_sock *smc,
+				   struct smc_roce_defs *roce_sugg,
+				   int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_confirm_msg *conf_llc;
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* receive CONFIRM LINK request over RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_CONFIRM_LINK);
+	if (!qentry) {
+		/* clc decl received or llc timeout */
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_NO_NOTIFY, smc);
+		return rc;
+	}
+	if (!(qentry->msg.hd.flags & 0xf)) {
+		/* no cache alignment specified, decline */
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		*reason_code = SMC_CLC_DEC_INT;
+		rc = -EOPNOTSUPP;
+		kfree(qentry);
+		return rc;
+	}
+	lgr->cache_align_size =
+		0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+	conf_llc = (struct llc_confirm_msg *)&qentry->msg;
+	link->link_id = conf_llc->link_num;
+	lgr->rtok[smc->conn.rtok_idx].link_id[smc->conn.lnk_idx] =
+		link->link_id;
+	lgr->max_links = SMC_MAX_SYM_LINKS;
+	kfree(qentry);
+	rc = smc_modify_qp_rts(link);
+	if (rc) {
+		*reason_code = SMC_CLC_DEC_INT;
+		return rc;
+	}
+	smc_check_qp_attr(link);
+	/* send CONFIRM LINK response over RoCE fabric */
+	rc = llc_send_confirm_link(link, (u8 *)link->roce_defs.mac,
+				   &roce_sugg->gid, LLC_RESP, LLC_SERV);
+	if (rc < 0) {
+		*reason_code = SMC_CLC_DEC_TCL;
+		return rc;
+	}
+	atomic_set(&link->state, SMC_LINK_UP);
+	return rc;
+}
+
+static int smc_clnt_add_lnk(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	release_sock(&smc->sk);
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_ADD_LINK);
+	lock_sock(&smc->sk);
+	if (!qentry) {
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_SERV, smc);
+		return rc;
+	}
+	rc = llc_cli_add_link(link, qentry, smc);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+	if (rc == -ENOLINK)
+		*reason_code = SMC_LLC_DEL_NOLNK;
+	return rc;
+}
+
+static void smc_save_peer_info(struct smc_sock *smc,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	smc->conn.local_tx_ctrl.ctrl_token = clc->rmb_alert_token;
+	smc->conn.peer_rx_buf_len =
+		smc_uncompress_bufsize(clc->flags2 & 0xf0);
+	smc->conn.peer_tcp_conn_idx = clc->tcp_conn_idx;
+	atomic_set(&smc->conn.rx_buf_space, smc->conn.peer_rx_buf_len);
+}
+
+static void smc_save_link_info(struct smc_link *link,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	memcpy(link->gid_peer, &clc->hdr2.gid, sizeof(link->gid_peer));
+	memcpy(link->mac_peer, clc->hdr2.mac, sizeof(link->mac_peer));
+	ntoh_three(&link->psn_peer, clc->psn);
+	link->mtu_peer = clc->flags2 & 0x0f;
+}
+
+static int smc_switch_to_tcp(struct smc_sock *smc, int reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	int rc;
+
+	smc->use_tcp = 1;
+	if (reason_code && (reason_code != SMC_CLC_DEC_REPLY)) {
+		rc = clc_send_decline(smc, reason_code, 0);
+		if (rc < sizeof(struct smc_decline_clc_msg))
+			return rc;
+	}
+	if (lgr) {
+		if (smc->new_lgr && link) {
+			atomic_set(&link->state, SMC_LINK_DOWN);
+			lgr->lgr_type = NONE;
+		}
+		if (lgr->lgr_type == NONE) {
+			spin_lock_bh(&smc_lgr_list.lock);
+			list_del_init(&lgr->list);
+			spin_unlock_bh(&smc_lgr_list.lock);
+		}
+	}
+	smc_free_conn(smc);
+	return 0;
+}
+
+static void smc_init_after_clc_handshake(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	u32 keepalive_time;
+
+	init_waitqueue_head(&smc->prep_waiter);
+	init_waitqueue_head(&smc->destruct_waiter);
+	if (sk->sk_state == SMC_INIT)
+		sk->sk_state = SMC_ACTIVE;
+	if (smc->use_tcp)
+		goto out;
+	INIT_DELAYED_WORK(&smc->write_work, smc_write_worker);
+	if (sock_flag(sk, SOCK_KEEPOPEN)) {
+		keepalive_time = tp->keepalive_time ? : TCP_KEEPALIVE_TIME;
+		if (keepalive_time)
+			sk_reset_timer(sk, &sk->sk_timer,
+				       jiffies + (unsigned long)keepalive_time);
+	}
+	sk->sk_data_ready = smc_sock_wake_rx;
+	sk->sk_write_space = smc_sock_wake_tx;
+	/* to return meaningful values with getsockopt */
+	sk->sk_sndbuf = smc->conn.rmb_tx_size;
+	sk->sk_rcvbuf = smc->conn.rmb_rx_size;
+out:
+	return;
+}
+
+static int smc_connect(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct socket *sock = sk->sk_socket;
+	int rc = 0;
+	struct sockaddr_in *inaddr =  (struct sockaddr_in *)smc->addr;
+	struct smc_acc_conf_clc_msg aclc;
+	struct smc_link_group *lgr = NULL;
+	struct smc_link *link = NULL;
+	struct smc_roce_defs roce_sugg;
+	u32 tmp_qpnum;
+	char pnet_id[SMC_MAX_PNET_ID_LEN];
+	int reason_code = 0;
+
+	smc_find_roce_resources(&roce_sugg, pnet_id, inaddr,
+				smc->tcpsocket->sk);
+	if (!roce_sugg.ibdev) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+	/* do inband token exchange */
+	rc = smc_send_proposal(smc, &roce_sugg, &aclc, &reason_code);
+	if (rc) {
+		if ((rc == -EPROTO) || (rc == -ETIMEDOUT)) {
+			smc->sk.sk_err = ENETUNREACH;
+			rc = -ENETUNREACH;
+		}
+		goto out_err;
+	}
+	if (reason_code) { /* switch to tcp */
+		smc->use_tcp = 1;
+		goto out_connected;
+	}
+
+	/* allocate connection/link group*/
+	ntoh_three(&tmp_qpnum, aclc.qp_num);
+	smc->new_lgr = aclc.hdr.flags & SMC_FIRST_CONTACT;
+	rc = smc_create_conn(smc, inaddr->sin_addr.s_addr, SMC_CLNT,
+			     &roce_sugg, &aclc.hdr2, tmp_qpnum);
+	if (rc == -ENOLINK) { /* link groups out of sync */
+		smc->sk.sk_err = ENETUNREACH;
+		rc = -ENETUNREACH;
+		goto out_err;
+	}
+	if (rc) {
+		/* insufficient memory */
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	lgr = smc->conn.lgr;
+	link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	strncpy(lgr->pnet_id, pnet_id, sizeof(pnet_id));
+	smc_save_peer_info(smc, &aclc);
+	lock_sock(sk);
+
+	rc = smc_create_rmbs(smc);
+	if (rc) {
+		/* insufficient memory */
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	atomic_set(&smc->conn.tx_buf_space, smc->conn.rmb_tx_size);
+	if (smc->new_lgr) {
+		rc = smc_initial_map_rmb(smc);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_RIM;
+			goto decline;
+		}
+		link->qp_num_peer = tmp_qpnum;
+		smc_save_link_info(link, &aclc);
+	}
+	rc = smc_rtoken_handling(lgr, smc, &aclc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_INT;
+		goto decline;
+	}
+	if (smc->new_lgr) {
+		rc = smc_ready_link(link);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_INT;
+			goto decline;
+		}
+		llc_initiate(lgr, LLC_GRP_ADD_LINK);
+		lgr->lgr_type = SINGLE;
+	} else { /* link already active */
+		rc = smc_announce_rmb(smc, &reason_code);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_TU;
+			goto decline;
+		}
+	}
+	rc = smc_send_confirm(smc);
+	if (rc)
+		goto out_err;
+
+	if (smc->new_lgr) {
+		/* QP confirmation over RoCE fabric */
+		rc = smc_clnt_conf_first_lnk(smc, &roce_sugg, &reason_code);
+		if (rc == -EINTR)
+			goto out_err;
+		if (rc)
+			goto decline;
+		/* receive ADD LINK request over RoCE fabric */
+		rc = smc_clnt_add_lnk(smc, &reason_code);
+		if (rc == -ENOLINK)
+			goto out_connected;
+		if (rc == -ETIMEDOUT)
+			goto out_err;
+		if (rc)
+			goto decline;
+	}
+
+out_connected:
+	sock->state = SS_CONNECTED;
+	smc_init_after_clc_handshake(smc);
+	/* copy setsockopt settings to tcpsocket to be prepared for fallback */
+	smc_copy_sockopt_settings(smc->tcpsocket->sk, sk);
+	/* to return meaningful values with getsockopt */
+	if (lgr) {
+		release_sock(sk);
+		if (mutex_is_locked(&lgr->conf_mutex))
+			mutex_unlock(&lgr->conf_mutex);
+	}
+	smc_apply_tmp_sockopts(sk->sk_socket, &smc->tmp_sockopts);
+	return 0;
+
+decline:
+	rc = smc_switch_to_tcp(smc, reason_code);
+	if (!rc)
+		goto out_connected;
+out_err:
+	if (lgr) {
+		if (smc->new_lgr)
+			lgr->lgr_type = NONE;
+		release_sock(sk);
+		if (mutex_is_locked(&lgr->conf_mutex))
+			mutex_unlock(&lgr->conf_mutex);
+		smc_free_conn(smc);
+	}
+	smc->use_tcp = 1;
+	sk->sk_state = SMC_CLOSED;
+	return rc;
+}
+
+static int smc_sock_connect(struct socket *sock, struct sockaddr *addr,
+			    int alen, int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc, loop = 0;
+
+	/* copy setsockopt settings to tcpsocket */
+	smc_copy_sockopt_settings(smc->tcpsocket->sk, sk);
+	/* set SMC capability on tcp-socket */
+	smc->addr = addr;
+	if (addr->sa_family == AF_SMC)
+		addr->sa_family = AF_INET;
+
+again:
+	rc = kernel_connect(smc->tcpsocket, addr, alen, flags);
+	if (rc == -ERESTARTSYS) {
+		msleep(1000);
+		loop++;
+		if (loop < 10)
+			goto again;
+		rc = -EAGAIN;
+	}
+	if (rc == -EINPROGRESS)
+		return rc;
+
+	if (!rc) {
+		rc = smc_connect(smc);
+		if (rc)
+			rc = -1;
+	} else {
+		smc->use_tcp = 1;
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_err = -rc;
+		sk->sk_state_change(sk);
+		rc = -1;
+	}
+
+	return rc;
+}
+
+static int smc_sock_listen(struct socket *sock, int backlog)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc;
+
+	lock_sock(sk);
+	/* copy setsockopt settings to tcpsocket */
+	smc_copy_sockopt_settings(smc->tcpsocket->sk, sk);
+	/* set SMC capability on tcp-socket */
+	rc = kernel_listen(smc->tcpsocket, backlog);
+
+	if (!rc) {
+		sk->sk_max_ack_backlog = backlog;
+		sk->sk_ack_backlog = 0;
+		sk->sk_state = SMC_LISTEN;
+		if (!smc->use_tcp) {
+			INIT_WORK(&smc->tcp_listen_work, smc_tcp_listen_worker);
+			schedule_work(&smc->tcp_listen_work);
+		}
+	}
+
+	release_sock(sk);
+	return rc;
+}
+
+static void smc_accept_enqueue(struct sock *parent, struct sock *sk)
+{
+	struct smc_sock *par = smc_sk(parent);
+
+	sock_hold(sk);
+	spin_lock(&par->accept_q_lock);
+	list_add_tail(&smc_sk(sk)->accept_q, &par->accept_q);
+	spin_unlock(&par->accept_q_lock);
+	sk_acceptq_added(parent);
+}
+
+static void smc_accept_unlink(struct sock *sk)
+{
+	struct smc_sock *par = smc_sk(sk)->listen_smc;
+
+	spin_lock(&par->accept_q_lock);
+	list_del_init(&smc_sk(sk)->accept_q);
+	spin_unlock(&par->accept_q_lock);
+	sk_acceptq_removed(&smc_sk(sk)->listen_smc->sk);
+	sock_put(sk);
+}
+
+static struct sock *smc_accept_dequeue(struct sock *parent,
+				       struct socket *newsock)
+{
+	struct smc_sock *isk, *n;
+	struct sock *sk;
+
+	list_for_each_entry_safe(isk, n, &smc_sk(parent)->accept_q, accept_q) {
+		sk = (struct sock *)isk;
+		lock_sock(sk);
+
+		smc_accept_unlink(sk);
+		if (newsock)
+			sock_graft(sk, newsock);
+		release_sock(sk);
+		return sk;
+	}
+	return NULL;
+}
+
+static int smc_sock_accept(struct socket *sock, struct socket *newsock,
+			   int flags)
+{
+	DECLARE_WAITQUEUE(wait, current);
+	struct sock *sk = sock->sk, *nsk;
+	long timeo;
+	int rc = 0;
+
+	lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+	timeo = sock_rcvtimeo(sk, flags & O_NONBLOCK);
+
+	/* Wait for an incoming connection */
+	add_wait_queue_exclusive(sk_sleep(sk), &wait);
+	while (!(nsk = smc_accept_dequeue(sk, newsock))) {
+		set_current_state(TASK_INTERRUPTIBLE);
+		if (!timeo) {
+			rc = -EAGAIN;
+			break;
+		}
+
+		release_sock(sk);
+		timeo = schedule_timeout(timeo);
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+
+		if (signal_pending(current)) {
+			rc = -EINTR;
+			break;
+		}
+	}
+
+	set_current_state(TASK_RUNNING);
+	remove_wait_queue(sk_sleep(sk), &wait);
+
+	if (rc) {
+		release_sock(sk);
+		goto done;
+	}
+	rc = sock_error(nsk);
+	if (rc) {
+		release_sock(sk);
+		goto done;
+	}
+
+	newsock->state = SS_CONNECTED;
+	release_sock(sk);
+	smc_apply_tmp_sockopts(newsock, &smc_sk(sk)->tmp_sockopts);
+
+	/* don't get value from tmp_sockopts as TCP will mangle the value */
+	timeo = 0;
+	if (smc_sk(sk)->tmp_sockopts.set & SMC_SOCKOPTS_DEFER_ACCEPT)
+		timeo = smc_getsockopt_from_tcp(smc_sk(nsk), TCP_DEFER_ACCEPT);
+
+	if (!rc && timeo && !(flags & O_NONBLOCK)) {
+		int rc = 0;
+
+		timeo = msecs_to_jiffies(timeo * 1000);
+		if (smc_sk(nsk)->use_tcp) {
+			lock_sock(smc_sk(nsk)->tcpsocket->sk);
+			rc = sk_wait_data(smc_sk(nsk)->tcpsocket->sk, &timeo,
+					  NULL);
+			release_sock(smc_sk(nsk)->tcpsocket->sk);
+		} else {
+			lock_sock(nsk);
+			rc = smc_wait_rx_data(smc_sk(nsk), 1, 1, timeo);
+			release_sock(nsk);
+		}
+	}
+done:
+	return rc;
+}
+
+static int smc_getsockopt_from_tcp(struct smc_sock *smc, int optname)
+{
+	char optval[10];
+	int optlen;
+	int rc;
+
+	optlen = sizeof(optlen);
+	memset(optval, 0, sizeof(optval));
+	rc = kernel_getsockopt(smc->tcpsocket, SOL_TCP, optname, optval,
+			       &optlen);
+	if (rc)
+		return -EINVAL;
+	memcpy(&rc, optval, optlen);
+	return rc;
+}
+
+static void smc_keepalive_timer(unsigned long);
+static void smc_sock_destruct(struct sock *);
+
+static int smc_send_accept(struct smc_sock *new_smc,
+			   struct smc_acc_conf_clc_msg *cclc, int *reason_code)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct smc_connection *conn = &new_smc->conn;
+	struct smc_acc_conf_clc_msg aclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int len, rc = 0;
+
+	link = &conn->lgr->lnk[conn->lnk_idx];
+	lgr = link->lnk_grp;
+	/* send SMC Accept CLC message (GID, MAC, VLAN, QP) */
+	memset(&aclc, 0, sizeof(struct smc_acc_conf_clc_msg));
+	aclc.hdr.flags = SMC_CLC_V1; /* smc version */
+	if (new_smc->new_lgr)
+		aclc.hdr.flags |= SMC_FIRST_CONTACT;
+	aclc.tcp_conn_idx = 1; /* as long as 1 RMB = 1 RMBE */
+	memcpy(&aclc.hdr2.gid, &link->roce_defs.gid,
+	       sizeof(link->roce_defs.gid));
+	memcpy(&aclc.hdr2.mac, link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+	hton_three(link->roce_qp->qp_num, aclc.qp_num);
+	hton_three(link->psn_initial, aclc.psn);
+	aclc.rmb_rkey =
+		htonl(conn->rmb_rx_elem->mr_rx[conn->lnk_idx]->rkey);
+	aclc.rmb_vaddr =
+		cpu_to_be64((u64)conn->rmb_rx_elem->rmb_rx_dma[conn->lnk_idx]);
+	memcpy(aclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(aclc.hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	aclc.hdr.type = SMC_CLC_ACCEPTANCE;
+	aclc.hdr.length = htons(sizeof(struct smc_acc_conf_clc_msg));
+	aclc.rmb_alert_token = conn->alert_token_local;
+	aclc.flags2 = conn->rmb_rx_size_short + link->path_mtu;
+	memcpy(aclc.ec_clcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &aclc;
+	vec.iov_len = sizeof(struct smc_acc_conf_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_sendmsg(new_smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_acc_conf_clc_msg));
+	if (len < sizeof(struct smc_acc_conf_clc_msg)) {
+		*reason_code = SMC_CLC_DEC_SEND;
+		goto out;
+	}
+
+	/* receive SMC Confirm CLC message (MAC, GID, QP, link #, link user id)
+	 */
+	rc = clc_wait_msg(new_smc, SMC_CLC_BLOCK, (char *)cclc,
+			  sizeof(struct smc_acc_conf_clc_msg),
+			  SMC_CLC_CONFIRMATION, reason_code);
+out:
+	return rc;
+}
+
+static int smc_serv_conf_first_lnk(struct smc_sock *smc,
+				   struct smc_roce_defs *roce_sugg,
+				   int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* send CONFIRM LINK request over the RoCE fabric */
+	llc_initiate(lgr, LLC_GRP_ADD_LINK);
+	rc = llc_send_confirm_link(link, (u8 *)link->roce_defs.mac,
+				   &roce_sugg->gid, LLC_REQ, LLC_SERV);
+	if (rc < 0) {
+		*reason_code = SMC_CLC_DEC_TCL;
+		lgr->lgr_type = NONE;
+		goto out;
+	}
+	/* receive CONFIRM LINK response over the RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_CONFIRM_LINK);
+	if (!qentry) {
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_SERV, smc);
+		atomic_set(&link->state, SMC_LINK_DOWN);
+		lgr->lgr_type = NONE;
+		if (rc == -ECOMM)
+			*reason_code = SMC_CLC_DEC_SEND;
+		if (!rc) /* clc decline received */
+			rc = -ECANCELED;
+	} else {
+		atomic_set(&link->state, SMC_LINK_UP);
+		if (!(qentry->msg.hd.flags & 0xf)) {
+			/* no cache alignment specified, decline */
+			kfree(qentry);
+			*reason_code = SMC_CLC_DEC_REPLY; /* no clc decline */
+			llc_send_del_link(link, link, LLC_FLAG_DEL_ORDERLY,
+					  LLC_DEL_PROT_VIOL, LLC_NO_NOTIFY);
+			atomic_set(&link->state, SMC_LINK_DOWN);
+			lgr->lgr_type = NONE;
+			schedule_delayed_work(&lgr->free_lgr_work,
+					      SMC_FREE_LGR_TIMEO);
+			rc = -EOPNOTSUPP;
+		} else {
+			lgr->cache_align_size = 0x00000001 <<
+				(((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+			kfree(qentry);
+		}
+	}
+out:
+	return rc;
+}
+
+static int smc_serv_add_lnk(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct smc_roce_defs roce_sugg;
+	int rc = 0;
+
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	roce_sugg.ibdev = NULL;
+	roce_sugg.port = 0;
+	roce_sugg.vlan = lgr->vlan;
+	smc_find_alt_roce_resources(&roce_sugg, lgr, link);
+	rc = llc_srv_add_link(link, &roce_sugg, smc);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+	return rc;
+}
+
+static void smc_listen_worker(struct work_struct *work)
+{
+	struct smc_sock *new_smc = container_of(work, struct smc_sock,
+						listen_work);
+	struct sock *newsmcsk = &new_smc->sk;
+	struct socket *newtcpsocket = new_smc->tcpsocket;
+	struct smc_sock *lsmc = new_smc->listen_smc;
+	struct sockaddr_in peeraddr;
+	char pclc_buf[sizeof(struct smc_proposal_clc_msg)
+		    + sizeof(struct smc_proposal_clc_msg2) + 40];
+	struct smc_proposal_clc_msg *pclc =
+		(struct smc_proposal_clc_msg *)&pclc_buf;
+	struct smc_proposal_clc_msg2 *pclc2;
+	struct smc_acc_conf_clc_msg cclc;
+	struct smc_link_group *lgr = NULL;
+	struct smc_link *link = NULL;
+	struct smc_roce_defs roce_sugg;
+	__be32 subnet = 0;
+	int reason_code = 0;
+	int rc = 0, len;
+	char pnet_id[SMC_MAX_PNET_ID_LEN];
+	u8 mlen = 0;
+
+	kernel_getpeername(newtcpsocket, (struct sockaddr *)&peeraddr, &len);
+	new_smc->new_lgr = 0;
+	/* do inband token exchange
+	 * receive SMC Proposal CLC message (peer ID, GID, MAC, VLAN)
+	 */
+	rc = clc_wait_msg(new_smc, SMC_CLC_BLOCK, (char *)&pclc_buf,
+			  sizeof(pclc_buf), SMC_CLC_PROPOSAL, &reason_code);
+	if (rc)
+		goto out_err;
+	if (reason_code)
+			goto decline;
+
+	smc_find_roce_resources(&roce_sugg, pnet_id, &peeraddr,
+				newtcpsocket->sk);
+	if (!roce_sugg.ibdev) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+
+	/* is it sufficient to use IP from connection itself? */
+	rc = smc_netinfo_by_tcpsk(&subnet, &mlen, newtcpsocket);
+	pclc2 = (struct smc_proposal_clc_msg2 *)
+		((char *)(pclc) + sizeof(struct smc_proposal_clc_msg)
+	       + pclc->iparea_offset);
+	if ((pclc2->outgoing_subnet != subnet) ||
+	    (pclc2->subnet_mask_bits != mlen)) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+
+	/* allocate connection/link group*/
+	rc = smc_create_conn(new_smc, peeraddr.sin_addr.s_addr,
+			     SMC_SERV, &roce_sugg, &pclc->hdr2, 0);
+	if (rc == -ENOLINK)
+		goto out_err;
+	if (rc) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	lgr = new_smc->conn.lgr;
+	link = &new_smc->conn.lgr->lnk[new_smc->conn.lnk_idx];
+
+	if (lgr->lgr_type == NONE) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	rc = smc_create_rmbs(new_smc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	atomic_set(&new_smc->conn.tx_buf_space, new_smc->conn.rmb_tx_size);
+
+	/* Announce RMB Rtoken */
+	if (new_smc->new_lgr) {
+		rc = smc_initial_map_rmb(new_smc);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_RIM;
+			goto decline;
+		}
+	} else {
+		rc = smc_announce_rmb(new_smc, &reason_code);
+		if (rc)
+			goto decline;
+	}
+	strncpy(lgr->pnet_id, pnet_id, sizeof(pnet_id));
+	rc = smc_send_accept(new_smc, &cclc, &reason_code);
+	if (rc)
+		goto out_err;
+	if (reason_code)
+		goto decline;
+
+	smc_save_peer_info(new_smc, &cclc);
+	if (new_smc->new_lgr) {
+		ntoh_three(&link->qp_num_peer, cclc.qp_num);
+		smc_save_link_info(link, &cclc);
+	}
+	rc = smc_rtoken_handling(lgr, new_smc,
+				 (struct smc_acc_conf_clc_msg *)&cclc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_INT;
+		goto decline;
+	}
+
+	if (new_smc->new_lgr) {
+		rc = smc_ready_link(link);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_INT;
+			goto decline;
+		}
+		lgr->max_links = SMC_MAX_SYM_LINKS;
+		lgr->lgr_type = SINGLE;
+		rc = smc_serv_conf_first_lnk(new_smc, &roce_sugg, &reason_code);
+		if ((rc == -EOPNOTSUPP) || (rc == -EINTR))
+			goto out_err; /* peer is not aware of a problem */
+		if (rc)
+			goto decline;
+		rc = smc_serv_add_lnk(new_smc, &reason_code);
+		if (rc == -ENOLINK)
+			goto out_connected;
+		else if (rc < 0)
+			goto out_err;
+	}
+
+out_connected:
+	if (lgr && mutex_is_locked(&lgr->conf_mutex))
+		mutex_unlock(&lgr->conf_mutex);
+	smc_sock_proc_create(newsmcsk);
+	sk_refcnt_debug_inc(newsmcsk);
+	if (!new_smc->use_tcp)
+		setup_timer(&newsmcsk->sk_timer, smc_keepalive_timer,
+			    (unsigned long)newsmcsk);
+	smc_init_after_clc_handshake(new_smc);
+enqueue:
+	if (lsmc->sk.sk_state == SMC_LISTEN) {
+		smc_accept_enqueue(&lsmc->sk, newsmcsk);
+	} else {
+		smc_destruct_non_accepted(newsmcsk);
+		new_smc = NULL;
+	}
+
+	/* Wake up accept */
+	lsmc->sk.sk_data_ready(&lsmc->sk);
+	sock_put(&lsmc->sk);
+	return;
+
+decline:
+	if (!smc_switch_to_tcp(new_smc, reason_code))
+		goto out_connected;
+
+out_err:
+	if (lgr && mutex_is_locked(&lgr->conf_mutex))
+		mutex_unlock(&lgr->conf_mutex);
+	smc_free_conn(new_smc);
+
+	if (!newsmcsk->sk_err)
+		newsmcsk->sk_err = EPROTO;
+	newsmcsk->sk_state = SMC_CLOSED;
+	if (newsmcsk->sk_err == ETIMEDOUT) {
+		/* forget the new socket */
+		if (new_smc->proc) {
+			proc_remove(new_smc->proc->proc);
+			new_smc->proc = NULL;
+		}
+		sock_set_flag(newsmcsk, SOCK_ZAPPED);
+		sock_set_flag(newsmcsk, SOCK_DEAD);
+		sock_release(newtcpsocket);
+		new_smc->tcpsocket = NULL;
+		sock_put(newsmcsk);
+		sock_put(&lsmc->sk);
+	} else {
+		goto enqueue;
+	}
+}
+
+static void smc_tcp_listen_worker(struct work_struct *work)
+{
+	struct smc_sock *lsmc = container_of(work, struct smc_sock,
+					tcp_listen_work);
+	struct sock *newsmcsk;
+	struct smc_sock *new_smc = NULL;
+	struct socket *newtcpsocket = NULL;
+	struct tcp_sock *ntp;
+	int rc = 0;
+
+	while (lsmc->sk.sk_state == SMC_LISTEN) {
+		rc = kernel_accept(lsmc->tcpsocket, &newtcpsocket, 0);
+		if (lsmc->sk.sk_state == SMC_CLOSED) {
+			if (newtcpsocket)
+				sock_release(newtcpsocket);
+			goto out;
+		}
+		if (rc) {
+			/* find out if there is an error we have to
+			 * restart tcp accept
+			 */
+			lsmc->sk.sk_err = -rc;
+			goto out;
+		}
+		ntp = tcp_sk(newtcpsocket->sk);
+
+		newsmcsk = smc_sock_alloc(NULL, PF_SMC, GFP_KERNEL);
+		if (!newsmcsk) {
+			lsmc->sk.sk_err = ENOMEM;
+			sock_release(newtcpsocket);
+			goto out;
+		}
+
+		new_smc = smc_sk(newsmcsk);
+		new_smc->tcpsocket = newtcpsocket;
+		new_smc->listen_smc = lsmc;
+		sock_hold(&lsmc->sk); /* sock_put in smc_listen_worker */
+		INIT_WORK(&new_smc->listen_work, smc_listen_worker);
+		smc_copy_sockopt_settings(newsmcsk, lsmc->tcpsocket->sk);
+		schedule_work(&new_smc->listen_work);
+	}
+
+out:
+	lsmc->sk.sk_data_ready(&lsmc->sk); /* wake up accept socket */
+}
+
+static int smc_sock_getname(struct socket *sock, struct sockaddr *addr,
+			    int *len, int peer)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc;
+
+	if (smc->tcpsocket)
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket, addr, len,
+						  peer);
+	else
+		rc = -EBADF;
+	return rc;
+}
+
+static int smc_sock_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int rc;
+
+	lock_sock(sk);
+	if (sk->sk_shutdown & SEND_SHUTDOWN) {
+		rc = -EPIPE;
+		goto out_pipe;
+	}
+
+	if (smc->use_tcp) {
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->sendmsg(smc->tcpsocket, msg,
+							  len);
+		else
+			rc = -EBADF;
+		goto out;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ECONNRESET;
+		goto out;
+	}
+	if (smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ENOTCONN;
+		goto out;
+	}
+	if (sk->sk_err) {
+		rc = -EPIPE;
+		goto out_pipe;
+	}
+
+	rc = smc_conn_send(smc, msg, len);
+out_pipe:
+	rc = sk_stream_error(sk, msg->msg_flags, rc);
+out:
+	release_sock(sk);
+
+	return rc;
+}
+
+static int smc_sock_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
+			    int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	lock_sock(sk);
+	if (smc->use_tcp) {
+		if (!smc->tcpsocket)
+			return -ENOTCONN;
+		rc = smc->tcpsocket->ops->recvmsg(smc->tcpsocket, msg, len,
+						  flags);
+	} else {
+		rc = smc_conn_recv(smc, msg, len, flags);
+	}
+	release_sock(sk);
+	return rc;
+}
+
+static unsigned int smc_accept_poll(struct sock *parent)
+{
+	struct smc_sock *isk;
+	struct sock *sk;
+
+	lock_sock(parent);
+	list_for_each_entry(isk, &smc_sk(parent)->accept_q, accept_q) {
+		sk = (struct sock *)isk;
+
+		if (sk->sk_state == SMC_ACTIVE) {
+			release_sock(parent);
+			return POLLIN | POLLRDNORM;
+		}
+	}
+	release_sock(parent);
+
+	return 0;
+}
+
+static unsigned int smc_sock_poll(struct file *file, struct socket *sock,
+				  poll_table *wait)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	unsigned int mask = 0;
+	int rc = 0;
+
+	if ((sk->sk_state == SMC_INIT) ||
+	    (smc->use_tcp && sk->sk_state != SMC_LISTEN)) {
+		if (smc->tcpsocket) {
+			mask = smc->tcpsocket->ops->poll(file, smc->tcpsocket,
+							 wait);
+			if ((sk->sk_state == SMC_INIT) && (mask & POLLOUT)) {
+				sk->sk_err = smc->tcpsocket->sk->sk_err;
+				if (sk->sk_err) {
+					mask |= POLLERR;
+				} else {
+					rc = smc_connect(smc);
+					if (rc)
+						mask |= POLLERR;
+					else
+						mask |= POLLOUT | POLLWRNORM;
+				}
+			}
+		} else {
+			sk->sk_err = EBADF;
+			mask |= POLLERR;
+		}
+	} else {
+		sock_poll_wait(file, sk_sleep(sk), wait);
+		if (sk->sk_state == SMC_LISTEN)
+			mask |= smc_accept_poll(sk);
+		if (sk->sk_err)
+			mask |= POLLERR;
+		if ((sk->sk_shutdown == SHUTDOWN_MASK) ||
+		    (sk->sk_state == SMC_CLOSED))
+			mask |= POLLHUP;
+		if (sk->sk_shutdown & RCV_SHUTDOWN)
+			mask |= POLLIN | POLLRDNORM | POLLRDHUP;
+		if (smc_to_read(smc))
+			mask |= POLLIN | POLLRDNORM;
+		if (sk->sk_state == SMC_APPLCLW1)
+			mask |= POLLIN;
+		if (!(sk->sk_shutdown & SEND_SHUTDOWN)) {
+			if (atomic_read(&smc->conn.tx_buf_space)) {
+				mask |= POLLOUT | POLLWRNORM;
+			} else {
+				set_bit(SOCK_ASYNC_NOSPACE,
+					&sk->sk_socket->flags);
+				set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+			}
+		} else {
+			mask |= POLLOUT | POLLWRNORM;
+		}
+		if ((smc->conn.rx_urg_state & SMC_URG_VALID) &&
+		    !(smc->conn.rx_urg_state & (SMC_URG_READ | SMC_URG_RECV)))
+			mask |= POLLPRI;
+	}
+	return mask;
+}
+
+static int smc_conn_shutdown(struct smc_sock *smc, int how)
+{
+	struct sock *sk = &smc->sk;
+	int rc = 0, ret = 0, old_state;
+
+	if (how == SHUT_RD)
+		return rc;
+
+	lock_sock(sk);
+	old_state = sk->sk_state;
+
+	if (how == SHUT_RDWR) {
+		smc_sk_release(smc, 1);
+		goto out;
+	}
+
+	/* how = SHUT_WR */
+	switch (sk->sk_state) {
+	/* Normal termination - Actice close parts */
+	case SMC_ACTIVE:
+		sk->sk_state = SMC_PEERCLW1;
+		/* fall through */
+	case SMC_PEERCLW1:
+		smc_wait_close(smc);
+		/* fall through */
+	case SMC_PEERCLW2:
+		/* if shutdown(both) and unread data then connection reset
+		 * if outstanding data to be written or RDMA not compl,
+		 * then mask conn as pending close
+		 */
+		smc->conn.local_tx_ctrl.conn_state_flags.sending_done = 1;
+		ret = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		break;
+	/* Normal termination - Passive close parts */
+	case SMC_APPLCLW1:
+		smc_wait_close(smc);
+		sk->sk_state = SMC_APPLCLW2;
+		smc->conn.local_tx_ctrl.conn_state_flags.sending_done = 1;
+		ret = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		break;
+	default:
+		break;
+	}
+
+out:
+	if (old_state != sk->sk_state)
+		sk->sk_state_change(&smc->sk);
+	release_sock(sk);
+	if (ret)
+		rc = -ENOTCONN;
+	return rc;
+}
+
+static int smc_sock_shutdown(struct socket *sock, int how)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (how > SHUT_RDWR)
+		return -EINVAL;
+
+	kernel_sock_shutdown(smc->tcpsocket, how);
+
+	if (!smc->use_tcp)
+		rc = smc_conn_shutdown(smc, how);
+	sk->sk_shutdown |= ++how;
+	return rc;
+}
+
+static u32 smc_keepalive_time_elapsed(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+
+	lgr = smc->conn.lgr;
+	link = &lgr->lnk[smc->conn.lnk_idx];
+	return (u32)jiffies - link->rcv_tstamp;
+}
+
+static void smc_set_keepalive(struct sock *sk, int valbool)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct sock *tp = smc->tcpsocket->sk;
+	u32 ka_tim;
+
+	tcp_set_keepalive(tp, valbool);
+	if (valbool)
+		sock_set_flag(tp, SOCK_KEEPOPEN);
+	else
+		sock_reset_flag(tp, SOCK_KEEPOPEN);
+	if (smc->use_tcp)
+		return;
+	if (sk->sk_state == SMC_ACTIVE) {
+		ka_tim = tcp_sk(tp)->keepalive_time ? : TCP_KEEPALIVE_TIME;
+		if (valbool && !sock_flag(sk, SOCK_KEEPOPEN)) {
+			sk_reset_timer(sk, &sk->sk_timer,
+				       jiffies + (unsigned long)ka_tim);
+		} else if (!valbool) {
+			sk_stop_timer(sk, &sk->sk_timer);
+			cancel_work_sync(&smc->timer_work);
+		}
+	}
+	if (valbool)
+		sock_set_flag(sk, SOCK_KEEPOPEN);
+	else
+		sock_reset_flag(sk, SOCK_KEEPOPEN);
+}
+
+static int smc_sock_setsockopt(struct socket *sock, int level, int optname,
+			       char __user *optval, unsigned int optlen)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct timer_list *cork_timer = &smc->cork_timer;
+	int val;
+	int valbool;
+	u32 ka_tim, elapsed = 0;
+	int rc = 0;
+
+	if (!smc->tcpsocket)
+		return -EBADF;
+	lock_sock(sk);
+	if (level != SOL_TCP)
+		goto fallback;
+	switch (optname) {
+	case SMC_KEEPALIVE:
+		if (get_user(val, (int __user *)optval)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		valbool = val ? 1 : 0;
+		smc_set_keepalive(sk, valbool);
+		break;
+	case TCP_KEEPIDLE:
+		rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket, SOL_TCP,
+			optname, optval, optlen);
+		if (rc || smc->use_tcp)
+			goto ret;
+		if (get_user(val, (int __user *)optval)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		ka_tim = val;
+		if (sk->sk_state == SMC_ACTIVE) {
+			if (ka_tim && sock_flag(sk, SOCK_KEEPOPEN)) {
+				elapsed = smc_keepalive_time_elapsed(smc);
+				if (ka_tim > elapsed)
+					elapsed = ka_tim - elapsed;
+				sk_reset_timer(sk, &sk->sk_timer,
+					       jiffies + elapsed);
+			} else {
+				sk_stop_timer(sk, &sk->sk_timer);
+				cancel_work_sync(&smc->timer_work);
+			}
+		}
+		break;
+	case TCP_CORK:
+		if (optlen < sizeof(int)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		if (get_user(val, (int __user *)optval)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			smc->tmp_sockopts.cork = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_CORK;
+		} else {
+			rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket,
+					level, optname, optval, optlen);
+			if (!rc && !smc->use_tcp) {
+				if (!val && timer_pending(cork_timer)) {
+					del_timer_sync(cork_timer);
+					destroy_timer_on_stack(cork_timer);
+					smc_write_data(smc);
+				}
+			}
+		}
+		break;
+	case TCP_DEFER_ACCEPT:
+		if (optlen < sizeof(int)) {
+			rc = -EINVAL;
+			goto ret;
+		}
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			if (get_user(val, (int __user *)optval)) {
+				rc = -EFAULT;
+				goto ret;
+			}
+			smc->tmp_sockopts.defer_accept = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_DEFER_ACCEPT;
+		} else {
+			goto fallback;
+		}
+		break;
+	case TCP_NODELAY:
+		if (optlen < sizeof(int)) {
+			rc = -EINVAL;
+			goto ret;
+		}
+		if (get_user(val, (int __user *)optval)) {
+			rc = -EFAULT;
+			goto ret;
+		}
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			smc->tmp_sockopts.nodelay = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_NODELAY;
+		} else {
+			if (!smc->use_tcp) {
+				if (val && timer_pending(cork_timer)) {
+					del_timer_sync(cork_timer);
+					destroy_timer_on_stack(cork_timer);
+					smc_write_data(smc);
+				}
+			}
+			goto fallback;
+		}
+		break;
+	default:
+		goto fallback;
+	}
+	goto ret;
+fallback:
+	rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket, level,
+		optname, optval, optlen);
+ret:
+	release_sock(sk);
+	return rc;
+}
+
+static int smc_sock_getsockopt(struct socket *sock, int level, int optname,
+			       char __user *optval, int __user *optlen)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int val, len;
+	int rc = 0;
+
+	if (smc->use_tcp)
+		if (!smc->tcpsocket)
+			return -EBADF;
+		return smc->tcpsocket->ops->getsockopt(smc->tcpsocket, level,
+						       optname, optval, optlen);
+	if (get_user(len, optlen))
+		return -EFAULT;
+	if (len < 0)
+		return -EINVAL;
+
+	if (level == SOL_IP) {
+		switch (optname) {
+		case IP_MTU:
+			if (sock->state != SS_CONNECTED || !smc->conn.lgr)
+				return -ENOTCONN;
+			val = ib_mtu_enum_to_int(
+				smc->conn.lgr->lnk[smc->conn.lnk_idx].mtu_peer);
+				break;
+		default:
+			goto fallback;
+		}
+	} else {
+		if (level != SOL_TCP)
+			goto fallback;
+	}
+	/* level is SOL_TCP */
+	switch (optname) {
+	case TCP_CORK:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = !!smc->tmp_sockopts.cork;
+		else
+			goto fallback;
+		break;
+	case TCP_DEFER_ACCEPT:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = smc->tmp_sockopts.defer_accept;
+		else
+			goto fallback;
+		break;
+	case TCP_NODELAY:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = !!smc->tmp_sockopts.nodelay;
+		else
+			goto fallback;
+		break;
+	default:
+		goto fallback;
+	}
+
+	if (len < sizeof(val) && len > 0 && val >= 0 && val <= 255) {
+		unsigned char ucval = (unsigned char)val;
+
+		len = 1;
+		if (put_user(len, optlen))
+			return -EFAULT;
+		if (copy_to_user(optval, &ucval, 1))
+			return -EFAULT;
+	} else {
+		len = min_t(unsigned int, sizeof(int), len);
+		if (put_user(len, optlen))
+			return -EFAULT;
+		if (copy_to_user(optval, &val, len))
+			return -EFAULT;
+	}
+
+	return rc;
+
+fallback:
+	return smc->tcpsocket->ops->getsockopt(smc->tcpsocket, level,
+					       optname, optval, optlen);
+}
+
+static int smc_sock_ioctl(struct socket *sock, unsigned int cmd,
+			  unsigned long arg)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int answ;
+	struct smc_curs urg, c_curs;
+
+	if (smc->use_tcp) {
+		if (!smc->tcpsocket)
+			return -EBADF;
+		return smc->tcpsocket->ops->ioctl(smc->tcpsocket, cmd, arg);
+	}
+
+	switch (cmd) {
+	case SIOCINQ:
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		if (sk->sk_state == SMC_INIT)
+			answ = 0;
+		else
+			answ = smc_to_read(smc);
+		break;
+	case SIOCATMARK:
+		if ((sk->sk_state == SMC_INIT) ||
+		    (sk->sk_state == SMC_LISTEN)) {
+			answ = 0;
+		} else {
+			struct smc_e_ctrl *tx_ctrl = &conn->local_tx_ctrl;
+
+			urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+			if (!sock_flag(&smc->sk, SOCK_URGINLINE))
+				smc_curs_add(conn->rmb_rx_size, &urg.s.curs, 1);
+
+			c_curs.s.lcurs =
+				atomic64_read(&tx_ctrl->c_curs.s.acurs);
+			if ((conn->rx_urg_state & SMC_URG_VALID) &&
+			    (urg.s.curs.c == c_curs.s.curs.c) &&
+			    (urg.s.curs.w == c_curs.s.curs.w))
+				answ = 1;
+			else
+				answ = 0;
+		}
+		break;
+	case SIOCOUTQ:
+		/* output queue size (not send + not acked) */
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		if (sk->sk_state == SMC_INIT)
+			answ = 0;
+		else
+			answ = smc->conn.rmb_tx_size -
+					atomic_read(&smc->conn.tx_buf_space);
+		break;
+	case SIOCOUTQNSD:
+		/* output queue size (not send only) for smc always 0 */
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		answ = 0;
+		break;
+	default:
+		return -ENOIOCTLCMD;
+	}
+
+	return put_user(answ, (int __user *)arg);
+}
+
+static ssize_t smc_sock_sendpage(struct socket *sock, struct page *page,
+				 int offset, size_t size, int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (smc->use_tcp)
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->sendpage(smc->tcpsocket, page,
+							   offset, size, flags);
+		else
+			rc = -EBADF;
+	else
+		rc = sock_no_sendpage(sock, page, offset, size, flags);
+
+	return rc;
+}
+
+static ssize_t smc_sock_splice_read(struct socket *sock, loff_t *ppos,
+				    struct pipe_inode_info *pipe, size_t len,
+				    unsigned int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (smc->use_tcp) {
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->splice_read(smc->tcpsocket,
+							      ppos, pipe, len,
+							      flags);
+		else
+			rc = -EBADF;
+	} else {
+		if (*ppos)
+			/* we don't support offsets on our socket */
+			return -ESPIPE;
+		lock_sock(sk);
+		rc = smc_conn_splice_read(smc, pipe, len, flags);
+		release_sock(sk);
+	}
+
+	return rc;
+}
+
+/* must look like tcp */
+static const struct proto_ops smc_sock_ops = {
+	.family		= PF_SMC,
+	.owner		= THIS_MODULE,
+	.release	= smc_sock_release,
+	.bind		= smc_sock_bind,
+	.connect	= smc_sock_connect,
+	.socketpair	= sock_no_socketpair,
+	.accept		= smc_sock_accept,
+	.getname	= smc_sock_getname,
+	.poll		= smc_sock_poll,
+	.ioctl		= smc_sock_ioctl,
+	.listen		= smc_sock_listen,
+	.shutdown	= smc_sock_shutdown,
+	.setsockopt	= smc_sock_setsockopt,
+	.getsockopt	= smc_sock_getsockopt,
+	.sendmsg	= smc_sock_sendmsg,
+	.recvmsg	= smc_sock_recvmsg,
+	.mmap		= sock_no_mmap,
+	.sendpage	= smc_sock_sendpage,
+	.splice_read	= smc_sock_splice_read,
+};
+
+static int smc_sock_init_tcp(struct smc_sock *smc)
+{
+	int rc;
+
+	rc = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
+			      &smc->tcpsocket);
+	smc->sk.sk_rcvbuf = max_t(size_t, smc_def_rcvbuf_size,
+					  smc->tcpsocket->sk->sk_rcvbuf);
+	smc->sk.sk_sndbuf = max_t(size_t, smc_def_sndbuf_size,
+					  smc->tcpsocket->sk->sk_sndbuf);
+	tcp_sk(smc->tcpsocket->sk)->nonagle |= TCP_NAGLE_OFF | TCP_NAGLE_PUSH;
+	return rc;
+}
+
+static void smc_timer_worker(struct work_struct *);
+
+static void smc_free_proc_worker(struct work_struct *work)
+{
+	struct smc_proc_dir_entry *proc_to_free =
+		container_of(work, struct smc_proc_dir_entry, free_proc_work);
+	proc_remove(proc_to_free->proc);
+	kfree(proc_to_free);
+}
+
+static void smc_sock_destruct(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+
+	if (sk->sk_state != SMC_CLOSED)
+		return;
+	sk->sk_state = SMC_DESTRUCT;
+	if (!sock_flag(sk, SOCK_DEAD))
+		return;
+
+	if (smc->proc) {
+		smc->proc->sk = NULL;
+		schedule_work(&smc->proc->free_proc_work);
+		smc->proc = NULL;
+	}
+
+	smc_free_conn(smc);
+
+	sk_refcnt_debug_dec(sk);
+}
+
+static void smc_release_worker(struct work_struct *work)
+{
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+					release_work);
+
+	lock_sock(&smc->sk); /* make sure smc_sock_release is done */
+	if (!smc->use_tcp)
+		smc_wait_for_pending_sends(smc);
+	release_sock(&smc->sk);
+	sk_stop_timer(&smc->sk, &smc->sk.sk_timer);
+	cancel_work_sync(&smc->timer_work);
+	del_timer_sync(&smc->fin_timer);
+	destroy_timer_on_stack(&smc->fin_timer);
+	sock_put(&smc->sk);
+}
+
+static void smc_timer_worker(struct work_struct *work)
+{
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+					timer_work);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct llc_qentry *qentry;
+	int i, probes, rest_time, lnk_idx;
+
+	lgr = smc->conn.lgr;
+	probes = tp->keepalive_probes ? : TCP_KEEPALIVE_PROBES;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		link = &lgr->lnk[i];
+		lnk_idx = link - &lgr->lnk[0];
+		if (atomic_read(&link->state) != SMC_LINK_UP)
+			continue;
+		/* initiate keepalive */
+		rest_time = llc_initiate(lgr, LLC_GRP_TEST_LINK);
+		if (!rest_time) {
+			link->probes_out++;
+			if ((link->probes_out >= probes) &&
+			    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0)) {
+				schedule_work(&lgr->link_down_work);
+			}
+			return;
+		}
+		if (llc_send_test_link(link, "KEEPALIVE_CHECK")) {
+			llc_stop(lgr, LLC_GRP_TEST_LINK);
+			return;
+		}
+		qentry = llc_wait(lgr, LLC_TESTLINK, LLC_WAIT_TIMER_TIMEO,
+				  LLC_TEST_LINK);
+		if (!qentry) {
+			link->probes_out++;
+			if ((link->probes_out >= probes) &&
+			    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0)) {
+				schedule_work(&lgr->link_down_work);
+			}
+		} else {
+			link->probes_out = 0;
+		}
+		kfree(qentry);
+		llc_stop(lgr, LLC_GRP_TEST_LINK);
+	}
+}
+
+static void smc_keepalive_timer(unsigned long data)
+{	/* derived from tcp_keepalive_timer() */
+	struct sock *sk = (struct sock *)data;
+	struct smc_sock *smc = smc_sk(sk);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct inet_connection_sock *icsk = inet_csk(smc->tcpsocket->sk);
+	u32 elapsed, time, intvl;
+
+	/* Only process if socket is not in use. */
+	bh_lock_sock(sk);
+	if (sock_owned_by_user(sk)) {
+		/* Try again later */
+		sk_reset_timer(sk, &sk->sk_timer, jiffies + (HZ / 20));
+		goto out;
+	}
+
+	if (!sock_flag(sk, SOCK_KEEPOPEN) || (sk->sk_state != SMC_ACTIVE))
+		goto out;
+
+	time = tp->keepalive_time ? : TCP_KEEPALIVE_TIME;
+	if (!time)
+		goto out;
+	intvl = tp->keepalive_intvl ? : TCP_KEEPALIVE_INTVL;
+	elapsed = smc_keepalive_time_elapsed(smc);
+
+	if ((elapsed >= intvl) &&
+	    (!icsk->icsk_user_timeout ||
+	     (elapsed >= icsk->icsk_user_timeout))) {
+		schedule_work(&smc->timer_work);
+		elapsed = intvl;
+	}
+
+	sk_reset_timer(sk, &sk->sk_timer, jiffies + (unsigned long)elapsed);
+out:
+	bh_unlock_sock(sk);
+	sock_put(sk); /* taken in sk_reset_timer() */
+}
+
+static void smc_corked_write_cb(unsigned long arg)
+{
+	struct smc_sock *smc = (struct smc_sock *)arg;
+
+	schedule_delayed_work(&smc->write_work, 0);
+}
+
+static void smc_fin_cb(unsigned long arg)
+{
+	struct smc_sock *smc = (struct smc_sock *)arg;
+	struct sock *tcpsk;
+
+	sock_hold(&smc->sk);
+	if (smc->sk.sk_state == SMC_CLOSED)
+		goto out;
+	smc->sk.sk_err = ETIMEDOUT;
+	smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+	if (smc->tcpsocket && smc->tcpsocket->sk) {
+		smc->tcpsocket->sk->sk_err = ECONNABORTED;
+		smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+	}
+	smc->sk.sk_state = SMC_CLOSED;
+	smc->sk.sk_shutdown = smc->sk.sk_shutdown | RCV_SHUTDOWN;
+	if (smc->tcpsocket) {
+		tcpsk = smc->tcpsocket->sk;
+		tcpsk->sk_shutdown = tcpsk->sk_shutdown | RCV_SHUTDOWN;
+	}
+	sock_set_flag(&smc->sk, SOCK_DEAD);
+	smc->sk.sk_state_change(&smc->sk);
+	wake_up_interruptible(&smc->rx_waiter);
+	smc_sock_wake_tx(&smc->sk);
+	smc_sock_wake_rx(&smc->sk);
+	if (cmpxchg(&smc->sock_put_done, 0, 1) == 0)
+		schedule_work(&smc->release_work);
+out:
+	sock_put(&smc->sk);
+}
+
+static struct sock *smc_sock_alloc(struct socket *sock, int proto, gfp_t prio)
+{
+	struct sock *sk;
+	struct smc_proc_dir_entry *pde = NULL;
+	struct smc_sock *smc;
+
+	pde = kzalloc(sizeof(*pde), GFP_KERNEL);
+	if (!pde)
+		return NULL;
+
+	sk = sk_alloc(&init_net, PF_SMC, prio, &smc_proto, 0);
+	if (!sk) {
+		kfree(pde);
+		return NULL;
+	}
+
+	pde->sk = sk;
+	smc = smc_sk(sk);
+	smc->tcpsocket = NULL;
+	smc->use_tcp = 0;
+	memset(&smc->conn, 0, sizeof(struct smc_connection));
+	smc->spd = NULL;
+	smc->proc = pde;
+
+	sock_init_data(sock, sk); /* sets sk_refcnt to 1 */
+	sk->sk_state = SMC_INIT;
+	INIT_LIST_HEAD(&smc->accept_q);
+	spin_lock_init(&smc->accept_q_lock);
+	sk->sk_destruct = smc_sock_destruct;
+	INIT_WORK(&smc->release_work, smc_release_worker);
+	INIT_WORK(&smc->timer_work, smc_timer_worker);
+	INIT_WORK(&smc->proc->free_proc_work, smc_free_proc_worker);
+	sock_reset_flag(sk, SOCK_ZAPPED);
+	sock_reset_flag(sk, SOCK_DONE);
+	sk->sk_protocol = proto;
+	init_waitqueue_head(&smc->splice_waiter);
+	setup_timer(&smc->cork_timer, smc_corked_write_cb, (unsigned long)smc);
+	setup_timer(&smc->fin_timer, smc_fin_cb, (unsigned long)smc);
+	smc_init_tmp_sockopts(smc);
+
+	return sk;
+}
+
+static int smc_sock_create(struct net *net, struct socket *sock, int protocol,
+			   int kern)
+{
+	struct sock *sk;
+	struct smc_sock *smc;
+	int rc = 0;
+
+	sock->state = SS_UNCONNECTED;
+
+	switch (sock->type) {
+	case SOCK_STREAM:
+		sock->ops = &smc_sock_ops;
+		break;
+	default:
+		return -ESOCKTNOSUPPORT;
+	}
+
+	sk = smc_sock_alloc(sock, protocol, GFP_KERNEL);
+	if (!sk)
+		return -ENOMEM;
+
+	smc = smc_sk(sk);
+	rc = smc_sock_init_tcp(smc);
+	if (rc) {
+		sk_free(sk);
+		smc = NULL;
+		goto out;
+	}
+	smc_sock_proc_create(sk);
+	setup_timer(&sk->sk_timer, smc_keepalive_timer, (unsigned long)sk);
+
+	sk_refcnt_debug_inc(sk);
+out:
+	return rc;
+}
+
+static const struct net_proto_family smc_sock_family_ops = {
+	.family	= AF_SMC,
+	.owner	= THIS_MODULE,
+	.create	= smc_sock_create,
+};
+
+static int __init af_smc_init(void)
+{
+	int rc;
+
+#ifdef CONFIG_PROC_FS
+	rc = smc_proc_init();
+	if (rc) {
+		rc = -EFAULT;
+		goto out;
+	}
+#else
+	pr_warn("smc requires /proc file system\n");
+	rc = -ENOENT;
+	goto out;
+#endif /* CONFIG_PROC_FS */
+
+	INIT_LIST_HEAD(&smc_ib_devices);
+	spin_lock_init(&smc_ib_dev_lock);
+	INIT_LIST_HEAD(&smc_lgr_list.list);
+	spin_lock_init(&smc_lgr_list.lock);
+	INIT_LIST_HEAD(&smc_lgr_freeing_list);
+	smc_lgr_list.create_conn_pending = 0;
+
+	rc = proto_register(&smc_proto, 0);
+	if (rc) {
+		pr_warn("smc proto_register fails with %d\n", rc);
+		goto out;
+	}
+
+	rc = sock_register(&smc_sock_family_ops);
+	if (rc) {
+		pr_warn("smc sock_register fails with %d\n", rc);
+		goto out_proto;
+	}
+
+	rc = ib_register_client(&smc_ib_client);
+	if (rc)
+		goto out_sock;
+
+	return 0;
+
+out_sock:
+	sock_unregister(PF_SMC);
+out_proto:
+	proto_unregister(&smc_proto);
+out:
+	return rc;
+}
+
+static void __exit af_smc_exit(void)
+{
+	ib_unregister_client(&smc_ib_client);
+	sock_unregister(PF_SMC);
+	proto_unregister(&smc_proto);
+	smc_proc_exit();
+}
+
+module_init(af_smc_init);
+module_exit(af_smc_exit);
+
+MODULE_AUTHOR("Ursula Braun <ursula.braun@de.ibm.com>");
+MODULE_DESCRIPTION("smc socket address family");
+MODULE_VERSION(VERSION);
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_NETPROTO(PF_SMC);
diff --git a/net/smc/af_smc.h b/net/smc/af_smc.h
new file mode 100644
index 0000000..1391cd8
--- /dev/null
+++ b/net/smc/af_smc.h
@@ -0,0 +1,706 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Definitions for the SMC module
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ */
+
+#include <linux/types.h>
+#include <net/tcp_states.h>
+
+extern struct list_head smc_ib_devices;
+extern struct ib_client smc_ib_client;
+
+struct smc_lgr_list_struct {
+	struct list_head list;
+	spinlock_t lock;	/* protects list of link groups */
+	u32 create_conn_pending;/* delay 2nd connection creation */
+};
+
+extern struct smc_lgr_list_struct smc_lgr_list; /* list of link groups */
+extern struct list_head smc_lgr_freeing_list; /* link groups to be freed */
+extern u32 smc_ctrl_buf_cnt;		/* # of ctrl buffers per link */
+extern u32 smc_max_conn_per_lgr;	/* max. # of connections per lgr */
+extern u32 smc_port_add_delay;		/* port add worker delay */
+extern atomic_t smc_lgr_num;		/* for creation of a unique lgr id */
+extern atomic_t smc_reconfiguring;	/* signal port adding / deleting */
+extern unsigned int smc_def_sndbuf_size;	/* minimum sndbuf size */
+extern unsigned int smc_def_rcvbuf_size;	/* minimum rcvbuf size */
+
+#define SMC_VERSION 0x10			/* first 4 bits */
+#define SMC_LOCAL_PEERID_RESET	"%%%%%%%%"
+
+#define SMC_MAX_PNET_ID_LEN	16
+
+#define SMC_FIRST_CONTACT 0x08			/* bit 4 */
+#define SMC_LINK_GROUP_OUT_OF_SYNCH 0x08	/* bit 4 */
+
+#define SMC_MAX_CQE 32768	/* max. # of completion queue elements */
+#define SMC_MAX_WRE 1024	/* max. # of work requests in flight per QP */
+#define SMC_MAX_POLL_CQE 10	/* max. # of compl. queue elements in 1 poll */
+#define SMC_MAX_SYM_LINKS 2	/* max. # of symmetric links */
+#define SMC_MAX_RMB 255		/* max. # of RMBs/connections per link group */
+#define SMC_MIN_RMBE_SIZE 16384	/* minimum size of an RMBE */
+#define SMC_MAX_PORT_ADD_DELAY 60	/* max. delay for port add (in secs) */
+
+#define SMC_RMBE_SIZE_VALUES	16
+
+#define SMC_LNK_EYE_CATCHER	"SMC_LINK"
+#define SMC_LNK_EYE_CATCH_LEN	8
+
+#define SMC_SIZE_OF_CTRL_DATA	44
+#define SMC_SIZE_OF_CTRL_BUF	sizeof(struct rmb_e_ctrl)
+
+#define SMC_URG_VALID		0x01
+#define SMC_URG_RECV		0x02
+#define SMC_URG_READ		0x04
+#define SMC_URG_MASK		0x03
+
+#define SMC_CLC_V1		0x10
+
+#define SMC_CLC_PROPOSAL	0x01
+#define SMC_CLC_ACCEPTANCE	0x02
+#define SMC_CLC_CONFIRMATION	0x03
+#define SMC_CLC_DECLINE		0x04
+
+#define SMC_CLC_BLOCK		0
+#define SMC_CLC_NONBLOCK	1
+
+#define SMC_QP_TIMEOUT			15 /* 4096 * 2 ** timeout usec */
+#define SMC_MIN_RNR_TIMER		5
+#define SMC_RETRY_CNT			7 /* 7: infinite */
+#define SMC_RNR_RETRY			7 /* 7: infinite */
+#define LLC_WAIT_FIRST_TIMEO		(5 * HZ)
+#define LLC_WAIT_TIMEO			(2 * HZ)
+#define LLC_WAIT_TIMER_TIMEO		HZ
+#define SMC_FREE_LGR_TIMEO		(600 * HZ)
+#define SMC_WAIT_PENDING_SENDS_TIMEO	(LLC_WAIT_TIMEO - 1 * HZ)
+#define SMC_WAIT_FREE_CTRL_BUF_TIMEO	HZ
+#define SMC_WAIT_RX_SPACE_TIMEO		(10 * HZ)
+#define SMC_WAIT_TX_SPACE_TIMEO		(10 * HZ)
+
+#define SMC_CLC_DEC_RUI		0x01000000  /* insufficient resources         */
+#define SMC_CLC_DEC_RIM		0x01010000  /* insufficient memory resources  */
+#define SMC_CLC_DEC_RIQ		0x01020000  /* insufficient QP resources      */
+#define SMC_CLC_DEC_TU		0x02000000  /* timeout                        */
+#define SMC_CLC_DEC_TPC		0x02010000  /* timeout w4 proposal            */
+#define SMC_CLC_DEC_TAS		0x02020000  /* timeout w4 accept              */
+#define SMC_CLC_DEC_TCC		0x02030000  /* timeout w4 confirm             */
+#define SMC_CLC_DEC_TCL		0x02040000  /* timeout w4 QP confirm          */
+#define SMC_CLC_DEC_CUC		0x03000000  /* configuration error            */
+#define SMC_CLC_DEC_CSLV	0x03010000  /* server lacks RoCE acc. to VLAN */
+#define SMC_CLC_DEC_CNIC	0x03020000  /* RNIC error                     */
+#define SMC_CLC_DEC_SU		0x04000000  /* synchronization error          */
+#define SMC_CLC_DEC_SFCB	0x04010000  /* first contact bit expected     */
+#define SMC_CLC_DEC_FORM	0x05000000  /* format error in received resp. */
+#define SMC_CLC_DEC_REPLY	0x06000000  /* reply to a received decline    */
+#define SMC_CLC_DEC_SEND	0x07000000  /* sending problem                */
+#define SMC_CLC_DEC_INT		0x99990000  /* internal error                 */
+
+#define SMC_LLC_DEL_NOLNK	0x00100000  /* Unknown Link ID (no link)      */
+#define SMC_LLC_DEL_NOLGR	0x00200000  /* Unknown Link Group             */
+
+#define SMC_SOCKOPTS_CORK		0x00000001
+#define SMC_SOCKOPTS_DEFER_ACCEPT	0x00000002
+#define SMC_SOCKOPTS_NODELAY		0x00000004
+
+#ifdef CONFIG_X86
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x01 /* cache_align_size 64 */
+#else
+#ifdef CONFIG_PPC
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x02 /* cache_align_size 128 */
+#else
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x03 /* cache_align_size 256 */
+#endif /* CONFIG_PPC */
+#endif /* CONFIG_X86 */
+
+#define smc_sk(__sk) ((struct smc_sock *)__sk)
+
+#define smc_lnk_downing(state) \
+	(atomic_cmpxchg(state, SMC_LINK_UP, SMC_LINK_DOWN) == SMC_LINK_UP)
+
+#define smc_stop_received(conn) \
+	(conn->local_rx_ctrl.conn_state_flags.sending_done || \
+	 conn->local_rx_ctrl.conn_state_flags.abnormal_close || \
+	 conn->local_rx_ctrl.conn_state_flags.closed_conn)
+
+#define smc_close_received(conn) \
+	(conn->local_rx_ctrl.conn_state_flags.abnormal_close || \
+	 conn->local_rx_ctrl.conn_state_flags.closed_conn)
+
+enum smc_role {		/* possible roles of a link group */
+	SMC_CLNT,	/* client */
+	SMC_SERV	/* server */
+};
+
+enum smc_state {	/* possible states of an SMC socket */
+	SMC_ACTIVE	= TCP_ESTABLISHED,	/* (1) connected */
+	SMC_INIT	= TCP_SYN_SENT,		/* (2) */
+	SMC_CLOSED	= TCP_CLOSE,		/* (7) */
+	SMC_LISTEN	= TCP_LISTEN,	/* (10)required for SO_ACCEPTCONN */
+	SMC_PEERCLW1	= 20,
+	SMC_PEERCLW2	= 21,
+	SMC_APPLCLW1	= 22,
+	SMC_APPLCLW2	= 23,
+	SMC_APPLFINCLW	= 24,
+	SMC_PEERFINCLW	= 25,
+	SMC_PEERABORTW	= 26,
+	SMC_PROCESSABORT = 27,
+	SMC_DESTRUCT	= 32
+};
+
+enum smc_link_state {	/* possible states of a link */
+	SMC_LINK_FREED,
+	SMC_LINK_DOWN,
+	SMC_LINK_ACTIVATING,
+	SMC_LINK_UP
+};
+
+enum llc_ctl_ind {	/* possible llc event control range */
+	LLC_CLNT,	/* as in smc_role */
+	LLC_SERV,	/* as in smc_role */
+	LLC_TESTLINK,
+	LLC_NO_NOTIFY
+};
+
+enum llc_msg_group {	/* classifies llc events */
+	LLC_GRP_NONE		= 0,
+	LLC_GRP_ADD_LINK	= 2,
+	LLC_GRP_DEL_LINK	= 4,
+	LLC_GRP_CONF_RKEY	= 6,
+	LLC_GRP_TEST_LINK	= 7
+};
+
+struct llc_ctl {	/* llc event data */
+	struct llc_qentry	*qentry;	/* llc event data */
+	enum llc_msg_group	active;		/* current llc event group */
+	int			lnk_idx;	/* active link for llc event */
+	int			ctrl_elem;	/* used ctrl buffer element */
+	int			wc_status;	/* status of sent llc event */
+};
+
+struct rmb_conn_state_flags {
+#if defined(__BIG_ENDIAN_BITFIELD)
+	u8	sending_done : 1;
+	u8	closed_conn : 1;
+	u8	abnormal_close : 1;
+	u8	reserved : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+	u8      reserved : 5;
+	u8      abnormal_close : 1;
+	u8      closed_conn : 1;
+	u8      sending_done : 1;
+#endif
+
+} __packed;
+
+struct rmb_producer_flags {
+#if defined(__BIG_ENDIAN_BITFIELD)
+	u8	 write_blocked : 1;	/* Writing Blocked, no rx buf space */
+	u8	 urg_data_pending : 1;	/* Urgent Data Pending */
+	u8	 urg_data_present : 1;	/* Urgent Data Present */
+	u8	 cons_curs_upd_req : 1;	/* cursor update requested */
+	u8	 last_msg_valid_req : 1;/* message replay due to failover */
+	u8	 reserved : 3;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+	u8       reserved : 3;
+	u8       last_msg_valid_req : 1;/* message replay due to failover */
+	u8       cons_curs_upd_req : 1; /* cursor update requested */
+	u8       urg_data_present : 1;  /* Urgent Data Present */
+	u8       urg_data_pending : 1;  /* Urgent Data Pending */
+	u8       write_blocked : 1;     /* Writing Blocked, no rx buf space */
+#endif
+} __packed;
+
+/* in host byte order */
+struct smc_cursor {		/* SMC cursor - an offset in an RMBE */
+	u8	reserved0;
+	u8	reserved;
+	u16	w;		/* window wrap sequence number */
+	u32	c;		/* cursor (= offset) part */
+} __aligned(8);
+
+struct smc_curs {		/* overlay for atomic cursor handling */
+	union {
+		struct smc_cursor	curs;
+		atomic64_t		acurs;
+		long long		lcurs;
+	} s;
+} __aligned(8);
+
+/* in host byte order */
+struct smc_e_ctrl {		/* RMBE control information */
+	u8				ctrl_type;	/* type = 0xFE */
+	u8				ctrl_len;	/* length = 44 */
+	u16				ctrl_seq;	/* connection seq # */
+	u32				ctrl_token;	/* alert_token */
+	struct smc_curs			p_curs;		/* producer cursor */
+	struct smc_curs			c_curs;		/* consumer cursor,
+							 * piggy backed ctrl
+							 */
+	struct rmb_producer_flags	p_flags;	/* conn. tx/rx status */
+	struct rmb_conn_state_flags	conn_state_flags; /* peer conn. status*/
+	u8				reserved[18];
+	u8				reserved1[4];	/* fill up to dw */
+} __packed __aligned(8);
+
+/* in network byte order */
+struct rmb_cursor {		/* SMC cursor */
+	u8	reserved0;
+	u8	reserved;
+	__be16	wwrap_seq;
+	__be32	count;
+} __aligned(8);
+
+struct rmb_curs {
+	union {
+		struct	rmb_cursor	curs;
+		atomic64_t		acurs;
+		long long		lcurs;
+	} r;
+} __aligned(8);
+
+/* in network byte order */
+struct rmb_e_ctrl {
+	u8				ctrl_type;
+	u8				ctrl_len;
+	__be16				ctrl_seq;
+	__be32				ctrl_token;
+	struct rmb_curs			p_curs;
+	struct rmb_curs			c_curs;
+	struct rmb_producer_flags	prod_flags;
+	struct rmb_conn_state_flags	conn_state_flags;
+	u8				reserved[18];
+	u8				reserved1[4];
+} __packed __aligned(8);
+
+enum lgr_type {		/* link status of a link group */
+	INIT,
+	SINGLE,			/* local and peer have just 1 ib-device */
+	ASYMMETRIC_LOCAL,	/* local has 1, peer has 2 ib-devices */
+	ASYMMETRIC_PEER,	/* local has 2, peer has 1 ib-device */
+	SYMMETRIC,		/* local and peer have 2 ib-devices */
+	NONE
+};
+
+struct pending_send {	/* control data for a pending send request */
+	struct smc_connection	*conn;		/* socket connection */
+	struct smc_curs		cursor;		/* tx RMBE cursor sent */
+	struct smc_curs		p_cursor;	/* rx RMBE cursor produced */
+	u64			wr_id_send;	/* work request id sent */
+	u16			ctrl_seq;	/* conn. tx sequence # */
+	u8			post_ctl;	/* llc event control index */
+	u8			reserved;
+};
+
+struct tx_sge {		/* scatter gather element for a send request */
+	u32 offset;
+	u32 len;
+};
+
+struct smc_rtoken {			/* address/key of remote RMB */
+	u64		vaddr[SMC_MAX_SYM_LINKS + 1];
+	u32		rkey[SMC_MAX_SYM_LINKS + 1];
+	atomic_t	active;			/* used / unused entry */
+	u8		link_id[SMC_MAX_SYM_LINKS + 1];		/* link ref. */
+	u8		reserved[7 - SMC_MAX_SYM_LINKS];
+};
+
+struct rmb_tx_addrs {			/* tx RMBE info */
+	struct list_head	list;
+	char			*rmb_tx_dma[SMC_MAX_SYM_LINKS + 1]; /* mapped */
+	char			*rmb_tx;	/* tx RMBE address */
+	u32			used;		/* currently used / unused */
+};
+
+struct rmb_rx_addrs {			/* rx RMBE info */
+	struct list_head	list;
+	char			*rmb_rx_dma[SMC_MAX_SYM_LINKS + 1]; /* mapped */
+	char			*rmb_rx;	/* rx RMBE address */
+	struct ib_mr		*mr_rx[SMC_MAX_SYM_LINKS + 1];	/* rkey */
+	u32			used;		/* currently used / unused */
+};
+
+struct smc_roce_defs {			/* ib-device infos for a link */
+	struct smc_ib_device	*ibdev;
+	union ib_gid		gid;
+	unsigned short		vlan;
+	char			mac[6];
+	u8			sgid_idx;
+	u8			port;
+};
+
+/* SMC Link - long living - for more than 1 SMC Connection */
+struct smc_link {
+	char			lnk_eyecatch[8];
+	struct smc_link_group	*lnk_grp;
+	struct ib_pd		*roce_pd;	/* IB protection domain,
+						 * unique for every ROCE QP
+						 */
+	struct ib_qp		*roce_qp;	/* IB queue pair */
+	struct ib_qp_attr	qp_attr;	/* IB queue pair attributes */
+	struct smc_roce_defs	roce_defs;	/* ib-device infos */
+	atomic_t		state;		/* link state */
+	enum ib_mtu		mtu_peer;	/* mtu size of peer */
+	enum ib_mtu		path_mtu;	/* used mtu */
+	u32			qp_num_peer;	/* QP # of peer */
+	char			mac_peer[6];	/* = gid[8:10] || gid[13:15] */
+	char			gid_peer[16];	/* gid of peer */
+	u8			link_id;	/* unique # within link group */
+
+	struct rmb_e_ctrl	*ctrl_buf_tx;	/* for send of WR_SENDs */
+	dma_addr_t		ctrl_dma_tx;	/* mapped addr for ctrl buff */
+	struct ib_send_wr	*send_wr;	/* send work requests area */
+	struct ib_mr		*mr_tx;		/* from ib_get_dma_mr */
+	struct ib_sge		*send_sge;	/* send scatter gather area */
+	struct pending_send	*pending;	/* in-flight sends area */
+	unsigned long		*pending_bit_mask; /* used ctrl_buf_tx elems  */
+	u64			wr_id_send;	/* seq # of last sent WQE */
+	u32			psn_initial;	/* starting send WQE seq # */
+	u32			send_wr_num;	/* # of send ctrl buffers */
+
+	struct rmb_e_ctrl	*ctrl_buf_rx;	/* for recv of WR_SENDs */
+	dma_addr_t		ctrl_dma_rx;	/* mapped addr for ctrl buff */
+	struct ib_recv_wr	*recv_wr;	/* recv work requests area */
+	struct ib_sge		*recv_sge;	/* recv scatter gather area */
+	u64			wr_id_recv;	/* seq # of last recv WQE */
+	u32			psn_peer;	/* starting recv WQE seq # */
+	u32			recv_wr_num;	/* # of recv ctrl buffers */
+	u32			rcv_tstamp;	/* time of last recv event */
+
+	wait_queue_head_t	wqe_waiter;	/* w4 free ctrl_buf_tx elem */
+	unsigned int		probes_out;	/* # of TESTLINK probes */
+};
+
+/* SMC Link Group */
+struct smc_link_group {
+	struct list_head	list;
+	enum smc_role		role;		/* client or server */
+	enum lgr_type		lgr_type;	/* link status for lgr */
+	u32			lgr_id;		/* unique lgr id */
+	__be32			daddr;		/* destination ip address */
+	__be32			subnet;	/* subnet mask */
+	u8			mask_len;	/* # of significant mask bits */
+	unsigned short		vlan;		/* vlan id of lgr */
+	char			pnet_id[SMC_MAX_PNET_ID_LEN];
+						/* pnet_id of ROCEs */
+	char			peer_peer_id[8];/* unique system id of peer */
+	u8			max_links;	/* allowed # of links (2) */
+	struct mutex		conf_mutex;	/* link added or removed */
+	struct rb_root		conns_all;	/* connection tree */
+	rwlock_t		conns_lock;	/* protects conns_all */
+	unsigned int		conns_num;	/* current # of connections */
+	struct list_head	rmb_tx_bufs[16]; /* tx buffers */
+	struct list_head	rmb_rx_bufs[16]; /* rx buffers */
+	struct smc_rtoken	rtok[SMC_MAX_RMB]; /* remote addr/key pairs */
+	struct smc_link		lnk[SMC_MAX_SYM_LINKS + 1]; /* links */
+	struct list_head	llc_event_q;	/* queue for llc events */
+	spinlock_t		llc_event_q_lock; /* protects llc_event_q */
+	struct llc_qentry	*delayed_q;	/* postpone llc event handl. */
+	struct work_struct	llc_event_work; /* llc event worker */
+	struct llc_ctl		llc_ctl[3];	/* llc event controller */
+	spinlock_t		llc_ctl_lock;	/* protects llc_ctl */
+	wait_queue_head_t	llc_waiter;	/* w4 next llc event */
+	struct workqueue_struct	*llc_wq;	/* work queue for llc events */
+	struct delayed_work	free_lgr_work;	/* delayed freeing of an lgr */
+	struct delayed_work	free_lgr_final_work; /* final lgr freeing */
+	struct smc_ib_device	*ibdev_up;	/* ib-device - new link */
+	struct work_struct	link_up_work;	/* adding link to lgr */
+	struct work_struct	link_down_work;	/* removing link from lgr */
+	struct work_struct	llc_add_link_work; /* llc add event worker */
+	struct work_struct	llc_del_link_work; /* llc del event worker */
+	struct work_struct	llc_conf_rkey_work;/* llc confirm rkey worker*/
+	struct work_struct	llc_del_rkey_work; /* llc del rkey worker */
+	atomic_t		link_num;	/* unique id last added link */
+	u32			lnk_up;		/* link up in progress */
+	u32			lnk_down[SMC_MAX_SYM_LINKS + 1];
+						/* link down in progress */
+	int			cache_align_size;	/* rdma_write align */
+	u8			port_up;	/* ib-device port - new link */
+	char			nwm_data[16];	/* network management data */
+	u8			nwm_flags;	/* network management flags */
+};
+
+struct smc_connection {
+	struct rb_node		alert_node;
+	struct smc_link_group	*lgr;		/* link group of connection */
+	u32			alert_token_local; /* created with custom logic
+						    * to address RMB and RMBE
+						    */
+	int			lnk_idx;	/* link currently used */
+
+	int			rtok_idx;	/* idx to peer rkey/addr */
+	int			peer_rx_buf_len; /* without eyecatcher */
+	u8			peer_tcp_conn_idx; /* from tcp handshake */
+
+	struct smc_e_ctrl	local_tx_ctrl;	/* preparation for sending */
+	struct smc_curs		local_tx_ctrl_fin; /* last completed target
+						    * offset
+						    */
+	struct smc_curs		tx_curs_prep;	/* tx - prepared data */
+	struct smc_curs		tx_curs_sent;	/* tx - sent data */
+	struct smc_curs		tx_curs_fin;	/* tx - confirmed by peer */
+	struct smc_curs		tx_urg_curs;	/* position of urgent byte */
+	atomic_t		tx_buf_space;	/* remaining space in tx rmb */
+	u16			tx_seq;		/* sequence # for WR SENDs */
+	u16			tx_seq_fin;	/* sequence # confirmed */
+	spinlock_t		send_lock;	/* protect wr_sends */
+	/* tx buffer */
+	struct rmb_tx_addrs	*rmb_tx_elem;	/* tx RMBE addresses */
+	int			rmb_tx_size;	/* tx RMBE size <== sock wmem */
+	int			rmb_tx_size_short;
+	int			tx_rmb_reused;	/* new / reused RMB */
+
+	struct smc_e_ctrl	local_rx_ctrl;	/* filled during event_handl. */
+	struct smc_curs		rx_curs_confirmed; /* confirmed to peer */
+	struct smc_curs		local_rx_spliced; /* pos of in-flight data */
+	struct smc_curs		rx_urg_curs;	/* cursor to urgent data */
+	u8			rx_urg_data;	/* 1 byte urgent data */
+	u8			rx_urg_state;	/* urg data avail / rcvd */
+	u16			ctrl_seq_rcvd;	/* seq # of last recv */
+	atomic_t		rx_buf_space;	/* remaining space in rx rmb */
+	atomic_t		bytes_to_rcv;	/* arrived data,
+						 * not yet received
+						 */
+	/* rx buffer */
+	struct rmb_rx_addrs	*rmb_rx_elem;
+	int			rmb_rx_size;	/* rx RMBE size <== sock rmem */
+	int			rmb_rx_size_short;
+	int			rx_rmb_reused;	/* new / reused TMB */
+	int			upd_cons_limit;	/* lower limit for consumer
+						 * cursor update
+						 */
+};
+
+struct smc_ib_device {				/* list of ib-devices */
+	struct list_head	list;
+	struct ib_device	*dev;
+	struct ib_device_attr	attr;		/* ib device attributes */
+	struct ib_port_attr	pattr[2];	/* ib dev. port attributes */
+	struct ib_event_handler	event_handler;	/* global ib_event handler */
+	struct ib_cq		*roce_cq_send;	/* send completion queue */
+	struct ib_cq		*roce_cq_recv;	/* recv completion queue */
+	struct tasklet_struct	recv_tasklet;
+	struct tasklet_struct	send_tasklet;
+	struct work_struct	port_err_work;	/* port error worker */
+	struct delayed_work	port_add_work;	/* port add worker */
+	u32			port_err[2];	/* port error triggered */
+	u32			port_add[2];	/* port add triggered */
+};
+
+/* eye catcher "SMCR" EBCDIC */
+static const char SMC_EC[] = {0xe2, 0xd4, 0xc3, 0xd9};
+
+struct smc_clc_msg_hdr {	/* header1 of clc messages */
+	u8 ec_clce[4];		/* eye catcher */
+	u8 type;		/* proposal / accept / confirm / decline */
+	__be16 length;
+	u8 flags;
+} __packed;
+
+struct smc_clc_msg_hdr2 {	/* header2 of clc messages */
+	u8 peer_id[8];		/* unique system id */
+	union ib_gid gid;
+	u8 mac[6];
+} __packed;
+
+struct smc_proposal_clc_msg {	/* proposal clc message */
+	struct smc_clc_msg_hdr hdr;
+	struct smc_clc_msg_hdr2 hdr2;
+	__u16 iparea_offset;	/* offset to IP address information area */
+} __packed;
+
+struct smc_proposal_clc_msg2 {
+	__be32 outgoing_subnet;	/* subnet mask */
+	u8 subnet_mask_bits;	/* number of significant bits in mask */
+	u8 reserved[2];
+	u8 n;			/* number of IPv6 prefixes in prefix array */
+	u8 ec_pclcf[4];		/* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_acc_conf_clc_msg {	/* accept / confirm clc message */
+	struct smc_clc_msg_hdr hdr;
+	struct smc_clc_msg_hdr2 hdr2;
+	u8 qp_num[3];		/* QP number */
+	__be32 rmb_rkey;	/* RMB rkey */
+	u8 tcp_conn_idx;	/* TCP connection index, which RMBE in RMB */
+	__be32 rmb_alert_token; /* unique connection id */
+	u8 flags2;		/* server's RMB buf size (compressed notation)
+				 * + QP mtu
+				 */
+	u8 reserved;
+	__be64 rmb_vaddr;	/* RMB virtual address */
+	u8 reserved2;
+	u8 psn[3];		/* initial packet sequence number */
+	u8 ec_clcf[4]; /* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_decline_clc_msg {
+	struct smc_clc_msg_hdr hdr;
+	u8 peer_id[8];		/* sender peer_id */
+	u32 peer_diagnosis;	/* diagnosis information */
+	u8 reserved2[4];
+	u8 ec_dclcf[4]; /* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_tmp_sockopts {
+	unsigned int	set;	/* indicates what sockopts were user-set */
+	int		cork;
+	int		defer_accept;
+	int		nodelay;
+};
+
+struct smc_proc_dir_entry {
+	struct sock *sk;
+	struct proc_dir_entry *proc;
+	struct work_struct free_proc_work;
+};
+
+struct smc_sock {				/* smc socket */
+	struct sock sk;
+	struct socket *tcpsocket;		/* internal tcp socket */
+	struct smc_connection conn;		/* connection */
+	struct sockaddr *addr;			/* inet connect address */
+	bool new_lgr;				/* new lgr created for conn. */
+	bool use_tcp;				/* fallback to tcp */
+	struct smc_sock *listen_smc;		/* listen parent */
+	struct work_struct tcp_listen_work;	/* handle tcp socket accepts */
+	struct work_struct listen_work;		/* prepare new accept socket */
+	struct list_head	accept_q;	/* sockets to be accepted */
+	spinlock_t		accept_q_lock;	/* protects accept_q */
+	struct work_struct release_work;	/* w4 finished sents before
+						 * closing socket finally
+						 */
+	wait_queue_head_t prep_waiter;		/* w4 prep sent when closing */
+	wait_queue_head_t destruct_waiter;	/* w4 pending sends */
+	struct timer_list	fin_timer;	/* ensure socket closing even
+						 * if peer is gone
+						 */
+	u32 sock_put_done;			/* allow race free closing */
+	struct work_struct timer_work;		/* keepalive TESTLINK sending */
+	struct delayed_work write_work;		/* RDMA writes of tx data */
+	wait_queue_head_t rx_waiter;		/* w4 space in rx RMBE */
+	wait_queue_head_t	splice_waiter;
+	struct splice_pipe_desc *spd;
+	struct timer_list	cork_timer;
+	struct smc_tmp_sockopts tmp_sockopts;
+	struct smc_proc_dir_entry *proc;
+	char proc_name[48];			/* sockname in /proc/net/smc */
+};
+
+int smc_create_conn(struct smc_sock *, __be32, enum smc_role,
+		    struct smc_roce_defs *, struct smc_clc_msg_hdr2 *, u32);
+void smc_find_roce_resources(struct smc_roce_defs *, char *,
+			     struct sockaddr_in *, struct sock *);
+void smc_find_alt_roce_resources(struct smc_roce_defs *,
+				 struct smc_link_group *, struct smc_link *);
+int smc_gid_by_dev(struct smc_roce_defs *);
+int smc_netinfo_by_tcpsk(__be32 *, u8 *, struct socket *);
+int smc_pnet_by_ibdev(u8, char *, struct smc_ib_device *);
+int smc_get_ib_mac(struct smc_roce_defs *);
+int smc_create_link(struct smc_link_group *, int, struct smc_roce_defs *);
+int smc_port_active(struct smc_ib_device *, u8);
+int smc_ready_link(struct smc_link *);
+void smc_link_up(struct smc_link_group *);
+void smc_link_down(struct smc_link_group *, int);
+int smc_find_rtoken_by_link(struct smc_link_group *, int, u32);
+int smc_modify_qp_rts(struct smc_link *);
+int smc_modify_qp_reset(struct smc_link *);
+int clc_wait_msg(struct smc_sock *, int, char *, int, u8, int *);
+int clc_send_decline(struct smc_sock *, u32, u8);
+
+int smc_create_rmbs(struct smc_sock *);
+int smc_del_rmbs(struct smc_sock *);
+int smc_map_rmbs_tx(struct smc_link_group *, int, int,
+		    struct rmb_tx_addrs *);
+int smc_map_rmbs_rx(struct smc_link_group *, int, int,
+		    struct rmb_rx_addrs *);
+long smc_get_dma_mr(struct smc_link_group *, int, struct rmb_rx_addrs *);
+long smc_map_rmbs(struct smc_sock *);
+long smc_map_rmbs_to_link(struct smc_link_group *, int, int);
+int smc_get_rtoken(struct smc_link_group *);
+u8 smc_compress_bufsize(int);
+int smc_uncompress_bufsize(u8);
+
+int smc_conn_send(struct smc_sock *, struct msghdr *, size_t);
+int smc_conn_recv(struct smc_sock *, struct msghdr *, size_t, int);
+int smc_to_read(struct smc_sock *);
+int smc_get_ctrl_buf(struct smc_link *, u32 *);
+int smc_wr_send(struct smc_link *, struct smc_connection *, u64, u32);
+int smc_wait_rx_data(struct smc_sock *, int, bool, long);
+void smc_sock_wake_tx(struct sock *);
+void smc_sock_wake_rx(struct sock *);
+void smc_send_tasklet_fn(unsigned long);
+void smc_recv_tasklet_fn(unsigned long);
+ssize_t smc_conn_splice_read(struct smc_sock *, struct pipe_inode_info *,
+			     size_t, unsigned int);
+void smc_write_worker(struct work_struct *);
+void smc_write_data(struct smc_sock *);
+
+int smc_check_peer_gone(struct smc_link_group *);
+void smc_free_conn(struct smc_sock *);
+void smc_remove_rx_rmb(struct smc_sock *);
+void smc_free_link(struct smc_link_group *, int);
+void smc_free_lgr(struct smc_link_group *);
+int smc_wait_no_pending_sends_on_link(struct smc_link *);
+int smc_ctrl_send(struct smc_connection *, int);
+int smc_send_close(struct smc_sock *);
+void smc_send_close_alt(struct smc_sock *);
+int smc_switch_conns(struct smc_link_group *, int, int);
+void smc_terminate_conn(struct smc_link_group *);
+void smc_conn_term(struct smc_sock *);
+int smc_prepared_sends(struct smc_connection *);
+int smc_pending_sends(struct smc_connection *);
+void smc_clear_pending_sends(struct smc_connection *);
+
+int smc_sock_proc_create(struct sock *);
+void smc_sock_proc_remove(char *);
+int smc_proc_init(void);
+void smc_proc_exit(void);
+void smc_check_dev_attr(struct smc_ib_device *);
+void smc_check_port_attr(struct smc_ib_device *, int);
+void smc_check_qp_attr(struct smc_link *);
+void smc_cq_handler_recv(struct ib_cq *, void *);
+void smc_cq_handler_send(struct ib_cq *, void *);
+void smc_cq_event_handler(struct ib_event *, void *);
+
+static inline void hton_three(u32 host, u8 *net)
+{
+	u32 t;
+
+	t = cpu_to_be32(host);
+	memcpy(net, ((u8 *)&t) + 1, 3);
+}
+
+static inline void ntoh_three(u32 *host, u8 *net)
+{
+	u32 t = 0;
+
+	memcpy(((u8 *)&t) + 1, net, 3);
+	*host = be32_to_cpu(t);
+}
+
+static inline void smc_curs_dec(int size, struct smc_cursor *curs)
+{
+	if (!curs->c)
+		curs->w--;
+	curs->c = (curs->c + size - 1) % size;
+}
+
+static inline void smc_curs_add(int size, struct smc_cursor *curs, int value)
+{
+	curs->w += (curs->c + value) / size;
+	curs->c = (curs->c + value) % size;
+}
+
+/* calculate cursor difference between old and new, where old <= new */
+static inline int smc_curs_diff(unsigned int size, struct smc_curs *old,
+				struct smc_curs *new)
+{
+	if (old->s.curs.w != new->s.curs.w)
+		return max_t(int, 0, ((size - old->s.curs.c) + new->s.curs.c));
+	return max_t(int, 0, (new->s.curs.c - old->s.curs.c));
+}
diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
new file mode 100644
index 0000000..4521b3f
--- /dev/null
+++ b/net/smc/smc_core.c
@@ -0,0 +1,3297 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Basic Transport Functions exploiting Infiniband API
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              Stefan Raspl <raspl@linux.vnet.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/wait.h>
+#include <linux/kthread.h>
+#include <linux/hardirq.h>
+#include <net/tcp.h>
+#include <net/sock.h>
+#include <linux/types.h>
+#ifdef CONFIG_S390
+#include <asm/debug.h>
+#endif
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <linux/splice.h>
+#include <linux/mm.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+#define LGR_RECONFIGURING 1
+
+/* Find the connection associated with the given alert token in the link group.
+ * Requires @conns_lock.
+ * @token	alert token to search for
+ * @lgr		 link group to search in
+ * Returns connection associated with token if found, NULL otherwise.
+ */
+static inline
+struct smc_connection *smc_find_conn(u32 token, struct smc_link_group *lgr)
+{
+	struct rb_node *node;
+	struct smc_connection *res = NULL;
+
+	node = lgr->conns_all.rb_node;
+	while (node) {
+		struct smc_connection *cur = rb_entry(node,
+					struct smc_connection, alert_node);
+
+		if (cur->alert_token_local > token) {
+			node = node->rb_left;
+		} else {
+			if (cur->alert_token_local < token) {
+				node = node->rb_right;
+			} else {
+				res = cur;
+				break;
+			}
+		}
+	}
+
+	return res;
+}
+
+/* Register connection's alert token in our lookup structure.
+ * Requires @conns_lock.
+ * @smc		connection to register
+ * Returns 0 on success, != otherwise.
+ */
+static void _smc_register_alert_token(struct smc_connection *conn)
+{
+	struct rb_node **link, *parent = NULL;
+	u32 token = conn->alert_token_local;
+
+	link = &conn->lgr->conns_all.rb_node;
+	while (*link) {
+		struct smc_connection *cur = rb_entry(*link,
+					struct smc_connection, alert_node);
+
+		parent = *link;
+		if (cur->alert_token_local > token)
+			link = &parent->rb_left;
+		else
+			link = &parent->rb_right;
+	}
+	/* Put the new node there */
+	rb_link_node(&conn->alert_node, parent, link);
+	rb_insert_color(&conn->alert_node, &conn->lgr->conns_all);
+}
+
+/* Register connection in link group by assigning an alert token
+ * registered in a search tree.
+ * Requires lgr->conns_lock being held.
+ * Note that '0' is a reserved value and not assigned.
+ */
+static void smc_register_in_lgr(struct smc_connection *conn)
+{
+	static atomic_t nexttoken = ATOMIC_INIT(0);
+
+	WARN_ON(conn->alert_token_local);
+	while (!conn->alert_token_local) {
+		conn->alert_token_local = atomic_inc_return(&nexttoken);
+		if (smc_find_conn(conn->alert_token_local, conn->lgr))
+			conn->alert_token_local = 0;
+	}
+	_smc_register_alert_token(conn);
+	conn->lgr->conns_num++;
+}
+
+/* Unregister and reset the alert token of the given connection
+ */
+static void smc_unregister_in_lgr(struct smc_connection *conn)
+{
+	struct smc_link_group *lgr = conn->lgr;
+
+	if (conn->alert_token_local) {
+		write_lock_bh(&lgr->conns_lock);
+		rb_erase(&conn->alert_node, &lgr->conns_all);
+		lgr->conns_num--;
+		write_unlock_bh(&lgr->conns_lock);
+	}
+}
+
+static void smc_sock_wake_rx_for_tx(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct socket *sock = sk->sk_socket;
+
+	if (sock && (sock->state == SS_CONNECTED) &&
+	    atomic_read(&smc->conn.rx_buf_space))
+		wake_up_interruptible(&smc->rx_waiter);
+}
+
+static void smc_free_rmbs(struct smc_connection *conn)
+{
+	int i;
+
+	if (conn->rmb_tx_elem) {
+		i = ror8(conn->rmb_tx_size_short, 4);
+		cmpxchg(&conn->rmb_tx_elem->used, 1, 0);
+	}
+	conn->rmb_tx_elem = NULL;
+	if (conn->rmb_rx_elem) {
+		i = ror8(conn->rmb_rx_size_short, 4);
+		cmpxchg(&conn->rmb_rx_elem->used, 1, 0);
+	}
+	conn->rmb_rx_elem = NULL;
+}
+
+static int smc_find_same_lnk(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (i == lnk_idx)
+			continue;
+		if ((lgr->lnk[i].roce_defs.ibdev ==
+			lgr->lnk[lnk_idx].roce_defs.ibdev) &&
+		    (lgr->lnk[i].roce_defs.port ==
+			lgr->lnk[lnk_idx].roce_defs.port))
+			return i;
+	}
+	return i;
+}
+
+static void smc_unmap_rmbs(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+	struct rmb_tx_addrs *txkp;
+	struct rmb_rx_addrs *rxkp;
+	struct smc_roce_defs *rocdefs = &lgr->lnk[lnk_idx].roce_defs;
+	int reuse_lnk_idx;
+
+	reuse_lnk_idx = smc_find_same_lnk(lgr, lnk_idx);
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(txkp, &lgr->rmb_tx_bufs[i], list) {
+			if (txkp->rmb_tx_dma[lnk_idx] &&
+			    (reuse_lnk_idx > SMC_MAX_SYM_LINKS))
+				ib_dma_unmap_single(rocdefs->ibdev->dev,
+					(dma_addr_t)txkp->rmb_tx_dma[lnk_idx],
+					smc_uncompress_bufsize(rol8(i, 4)),
+					DMA_TO_DEVICE);
+			txkp->rmb_tx_dma[lnk_idx] = NULL;
+		}
+		list_for_each_entry(rxkp, &lgr->rmb_rx_bufs[i], list) {
+			if (rxkp->rmb_rx_dma[lnk_idx] &&
+			    (reuse_lnk_idx > SMC_MAX_SYM_LINKS))
+				ib_dma_unmap_single(rocdefs->ibdev->dev,
+					(dma_addr_t)rxkp->rmb_rx_dma[lnk_idx],
+					smc_uncompress_bufsize(rol8(i, 4)),
+					DMA_FROM_DEVICE);
+			rxkp->rmb_rx_dma[lnk_idx] = NULL;
+			if (rxkp->mr_rx[lnk_idx])
+				ib_dereg_mr(rxkp->mr_rx[lnk_idx]);
+			rxkp->mr_rx[lnk_idx] = NULL;
+		}
+	}
+}
+
+static void smc_clear_rtokens(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		lgr->rtok[i].rkey[lnk_idx] = 0;
+		lgr->rtok[i].vaddr[lnk_idx] = 0;
+		lgr->rtok[i].link_id[lnk_idx] = 0;
+	}
+}
+
+static void smc_free_link_mem(struct smc_link *lnk)
+{
+	kfree(lnk->pending);
+	lnk->pending = NULL;
+	kfree(lnk->pending_bit_mask);
+	lnk->pending_bit_mask = NULL;
+	kfree(lnk->send_sge);
+	lnk->send_sge = NULL;
+	kfree(lnk->recv_sge);
+	lnk->recv_sge = NULL;
+	kfree(lnk->recv_wr);
+	lnk->recv_wr = NULL;
+	kfree(lnk->send_wr);
+	lnk->send_wr = NULL;
+	kfree(lnk->ctrl_buf_tx);
+	lnk->ctrl_buf_tx = NULL;
+	kfree(lnk->ctrl_buf_rx);
+	lnk->ctrl_buf_rx = NULL;
+}
+
+static int smc_modify_qp_init(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_INIT;
+	qp_attr.pkey_index = 0;
+	qp_attr.port_num = link->roce_defs.port;
+	qp_attr.qp_access_flags = IB_ACCESS_LOCAL_WRITE
+				| IB_ACCESS_REMOTE_WRITE;
+	rc = ib_modify_qp(link->roce_qp, &qp_attr,
+			  IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_ACCESS_FLAGS |
+			  IB_QP_PORT);
+	return rc;
+}
+
+static int smc_modify_qp_rtr(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+	enum ib_qp_attr_mask qp_attr_mask =
+		IB_QP_STATE | IB_QP_AV | IB_QP_PATH_MTU | IB_QP_DEST_QPN |
+		IB_QP_RQ_PSN | IB_QP_MAX_DEST_RD_ATOMIC | IB_QP_MIN_RNR_TIMER |
+		IB_QP_SMAC;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RTR;
+	qp_attr.path_mtu = min(link->path_mtu, link->mtu_peer);
+	qp_attr.ah_attr.port_num = link->roce_defs.port;
+	qp_attr.ah_attr.ah_flags = IB_AH_GRH;
+	qp_attr.ah_attr.grh.hop_limit = 1;
+	memcpy(&qp_attr.ah_attr.grh.dgid, link->gid_peer,
+	       sizeof(link->gid_peer));
+	qp_attr.ah_attr.grh.sgid_index = link->roce_defs.sgid_idx;
+	if (link->roce_defs.vlan) {
+		qp_attr.ah_attr.vlan_id = link->roce_defs.vlan;
+		qp_attr.vlan_id = link->roce_defs.vlan;
+		qp_attr_mask |= IB_QP_VID;
+	}
+	memcpy(&qp_attr.ah_attr.dmac, link->mac_peer,
+	       sizeof(link->mac_peer));
+	qp_attr.dest_qp_num = link->qp_num_peer;
+	qp_attr.rq_psn = link->psn_peer; /* starting receive packet seq # */
+	qp_attr.max_dest_rd_atomic = 1;  /* max # of resources for incoming
+					  * requests
+					  */
+	qp_attr.min_rnr_timer = SMC_MIN_RNR_TIMER;
+	memcpy(&qp_attr.smac, link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+
+	rc = ib_modify_qp(link->roce_qp, &qp_attr, qp_attr_mask);
+	return rc;
+}
+
+int smc_modify_qp_rts(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RTS;
+	qp_attr.timeout = SMC_QP_TIMEOUT;	/* local ack timeout */
+	qp_attr.retry_cnt = SMC_RETRY_CNT;	/* retry count */
+	qp_attr.rnr_retry = SMC_RNR_RETRY;	/* RNR retries, 7=infinite */
+	qp_attr.sq_psn = link->psn_initial;	/* starting send packet seq # */
+	qp_attr.max_rd_atomic = 1;	/* # of outstanding RDMA reads and
+					 * atomic ops allowed
+					 */
+	rc = ib_modify_qp(link->roce_qp, &qp_attr,
+			  IB_QP_STATE | IB_QP_TIMEOUT | IB_QP_RETRY_CNT |
+			  IB_QP_SQ_PSN | IB_QP_RNR_RETRY |
+			  IB_QP_MAX_QP_RD_ATOMIC);
+	return rc;
+}
+
+int smc_modify_qp_reset(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RESET;
+	rc = ib_modify_qp(link->roce_qp, &qp_attr, IB_QP_STATE);
+	return rc;
+}
+
+static inline int smc_outstanding_post_recv(struct smc_link *link)
+{
+	int rc = 0;
+	struct ib_recv_wr *bad_recv_wr = NULL;
+	u32 index;
+	u64 wr_id;
+
+	wr_id = link->wr_id_recv++; /* tasklet context, thus not atomic */
+	index = wr_id % link->recv_wr_num;
+	link->recv_wr[index].wr_id = wr_id;
+	rc = ib_post_recv(link->roce_qp, &link->recv_wr[index], &bad_recv_wr);
+	return rc;
+}
+
+int smc_ready_link(struct smc_link *link)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	int rc = 0;
+	u32 i;
+
+	rc = smc_modify_qp_init(link);
+	if (rc)
+		goto out;
+
+	rc = smc_modify_qp_rtr(link);
+	if (rc)
+		goto out;
+	smc_check_qp_attr(link);
+	rc = ib_req_notify_cq(link->roce_defs.ibdev->roce_cq_recv,
+			      IB_CQ_SOLICITED_MASK);
+	if (rc)
+		goto out;
+	for (i = 0; i < link->recv_wr_num; i++)
+		rc = smc_outstanding_post_recv(link);
+	smc_check_qp_attr(link);
+
+	if (lgr->role == SMC_SERV) {
+		rc = smc_modify_qp_rts(link);
+		if (rc)
+			goto out;
+		smc_check_qp_attr(link);
+	}
+out:
+	return rc;
+}
+
+static void smc_free_spd(struct smc_sock *);
+
+/* reset rx_rmb if rkey exchange fails */
+void smc_remove_rx_rmb(struct smc_sock *smc)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct rmb_rx_addrs *rxkp = conn->rmb_rx_elem;
+	int lnk_idx = conn->lnk_idx;
+	struct smc_link *lnk = &conn->lgr->lnk[lnk_idx];
+	struct smc_roce_defs *rocdefs = &lnk->roce_defs;
+
+	if (rxkp->mr_rx[lnk_idx])
+		ib_dereg_mr(rxkp->mr_rx[lnk_idx]);
+	if (rxkp->rmb_rx_dma[lnk_idx])
+		ib_dma_unmap_single(rocdefs->ibdev->dev,
+				    (dma_addr_t)rxkp->rmb_rx_dma[lnk_idx],
+				    conn->rmb_rx_size, DMA_FROM_DEVICE);
+	rxkp->rmb_rx_dma[lnk_idx] = NULL;
+	list_del(&rxkp->list);
+	kfree(rxkp);
+	conn->rmb_rx_elem = NULL;
+}
+
+void smc_free_conn(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int empty;
+
+	if (lgr) {
+		/* can be called from sk_free in atomic context */
+		smc_clear_pending_sends(&smc->conn);
+		smc_free_rmbs(&smc->conn);
+		smc_unregister_in_lgr(&smc->conn);
+		empty = RB_EMPTY_ROOT(&smc->conn.lgr->conns_all);
+		if (empty)
+			schedule_delayed_work(&lgr->free_lgr_work,
+					      SMC_FREE_LGR_TIMEO);
+		smc->conn.lgr = NULL;
+	}
+	if (smc->spd)
+		smc_free_spd(smc);
+}
+
+void smc_free_link(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct smc_link *lnk = &lgr->lnk[lnk_idx];
+	struct smc_ib_device *ibdev = lnk->roce_defs.ibdev;
+	enum smc_link_state old_state;
+
+	old_state = atomic_cmpxchg(&lnk->state, SMC_LINK_DOWN, SMC_LINK_FREED);
+	if (old_state != SMC_LINK_DOWN) {
+		if (old_state == SMC_LINK_FREED) /* link freeing in progress */
+			while (lnk->roce_defs.ibdev)
+				msleep(10000);
+		return;
+	}
+	lnk->link_id = 0;
+	memset(lnk->lnk_eyecatch, 0, SMC_LNK_EYE_CATCH_LEN);
+	memset(lnk->pending_bit_mask, 0,
+	       BITS_TO_LONGS(smc_ctrl_buf_cnt) * sizeof(unsigned long));
+	if (!lnk->roce_defs.ibdev)
+		return;
+	if (lnk->roce_qp) {
+		ib_destroy_qp(lnk->roce_qp);
+		lnk->roce_qp = NULL;
+	}
+	smc_unmap_rmbs(lgr, lnk_idx);
+	memset(&lnk->roce_defs, 0, sizeof(struct smc_roce_defs));
+	lnk->qp_num_peer = 0;
+	smc_clear_rtokens(lgr, lnk_idx);
+	if (lnk->mr_tx) {
+		ib_dereg_mr(lnk->mr_tx);
+		lnk->mr_tx = NULL;
+	}
+	if (lnk->ctrl_dma_rx) {
+		ib_dma_unmap_single(ibdev->dev, lnk->ctrl_dma_rx,
+				(sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+				    DMA_FROM_DEVICE);
+		lnk->ctrl_dma_rx = 0;
+	}
+	if (lnk->ctrl_dma_tx) {
+		ib_dma_unmap_single(ibdev->dev, lnk->ctrl_dma_tx,
+				(sizeof(struct rmb_e_ctrl) * lnk->send_wr_num),
+				    DMA_TO_DEVICE);
+		lnk->ctrl_dma_tx = 0;
+	}
+	if (lnk->roce_pd) {
+		ib_dealloc_pd(lnk->roce_pd);
+		lnk->roce_pd = NULL;
+	}
+}
+
+static int smc_pending_sends_on_link(struct smc_link *link)
+{
+	u32 i;
+
+	i = find_first_bit(link->pending_bit_mask, link->send_wr_num);
+	if (i < link->send_wr_num)
+		return 1;
+	return 0;
+}
+
+int smc_wait_no_pending_sends_on_link(struct smc_link *link)
+{
+	int rc = 1;
+
+	if (smc_pending_sends_on_link(link)) {
+		rc = wait_event_interruptible_timeout(link->wqe_waiter,
+			!smc_pending_sends_on_link(link),
+			SMC_WAIT_PENDING_SENDS_TIMEO);
+	}
+	return rc;
+}
+
+void smc_free_lgr(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry, *q;
+	struct rmb_tx_addrs *entrytx, *tmptx;
+	struct rmb_rx_addrs *entryrx, *tmprx;
+	int i, rc;
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_del_init(&lgr->list); /* remove from smc_lgr_freeing_list */
+	spin_unlock_bh(&smc_lgr_list.lock);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (smc_lnk_downing(&lgr->lnk[i].state))
+			rc = smc_wait_no_pending_sends_on_link(&lgr->lnk[i]);
+		smc_free_link(lgr, i);
+	}
+	cancel_delayed_work_sync(&lgr->free_lgr_work);
+	cancel_work_sync(&lgr->llc_add_link_work);
+	if (lgr->role == SMC_SERV)
+		cancel_work_sync(&lgr->llc_del_link_work);
+	cancel_work_sync(&lgr->llc_conf_rkey_work);
+	cancel_work_sync(&lgr->llc_del_rkey_work);
+	destroy_workqueue(lgr->llc_wq);
+	spin_lock_bh(&lgr->llc_event_q_lock);
+	list_for_each_entry_safe(qentry, q, &lgr->llc_event_q, list) {
+		list_del_init(&qentry->list);
+		kfree(qentry);
+	}
+	spin_unlock_bh(&lgr->llc_event_q_lock);
+	list_for_each_entry_safe(entrytx, tmptx, &lgr->rmb_tx_bufs[i], list) {
+		list_del(&entrytx->list);
+		kfree(entrytx->rmb_tx);
+		kfree(entrytx);
+	}
+	list_for_each_entry_safe(entryrx, tmprx, &lgr->rmb_rx_bufs[i], list) {
+		list_del(&entryrx->list);
+		kfree(entryrx->rmb_rx);
+		kfree(entryrx);
+	}
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		smc_free_link_mem(&lgr->lnk[i]);
+		lgr->lnk[i].lnk_grp = NULL;
+	}
+	kfree(lgr);
+}
+
+static void smc_free_lgr_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  free_lgr_work.work);
+	struct llc_qentry *qentry;
+	int i, rest_time, rc, peer_notified = 0, peer_gone;
+
+	if (list_empty(&lgr->list))
+		peer_gone = -EPIPE;
+	else
+		peer_gone = smc_check_peer_gone(lgr);
+	if ((lgr->role == SMC_CLNT) && !peer_gone)
+		return;		/* server to trigger lgr removal */
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	read_lock_bh(&lgr->conns_lock);
+	rc = RB_EMPTY_ROOT(&lgr->conns_all);
+	read_unlock_bh(&lgr->conns_lock);
+	if (!rc) {
+		spin_unlock_bh(&smc_lgr_list.lock);
+		return;
+	}
+	if (atomic_read(&smc_reconfiguring) ||
+	    mutex_is_locked(&lgr->conf_mutex)) {
+		/* postpone freeing */
+		spin_unlock_bh(&smc_lgr_list.lock);
+		schedule_delayed_work(&lgr->free_lgr_work,
+				      LLC_WAIT_FIRST_TIMEO);
+		return;
+	}
+	list_move_tail(&lgr->list, &smc_lgr_freeing_list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+	if (peer_gone)
+		goto out;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (!smc_lnk_downing(&lgr->lnk[i].state) ||
+		    (lgr->lgr_type == NONE) ||
+		    peer_notified)
+			continue;
+		rest_time = llc_initiate(lgr, LLC_GRP_DEL_LINK);
+		if (!rest_time) {		/* timeout */
+			llc_stop(lgr, LLC_GRP_DEL_LINK);
+			continue;
+		}
+		if (!&lgr->lnk[i].link_id) {	/* link already freed */
+			llc_stop(lgr, LLC_GRP_DEL_LINK);
+			continue;
+		}
+		atomic_set(&lgr->lnk[i].state, SMC_LINK_UP); /* temporarily */
+		rc = llc_send_del_link(&lgr->lnk[i], NULL, LLC_FLAG_DEL_ORDERLY,
+				       LLC_DEL_PROG_INIT_TERM, LLC_SERV);
+		if (rc >= 0) {
+			lgr->llc_ctl[LLC_SERV].qentry = NULL;
+			qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO,
+					  LLC_DEL_LINK);
+			if (qentry)
+				peer_notified = 1;
+		}
+		llc_stop(lgr, LLC_GRP_DEL_LINK);
+		atomic_set(&lgr->lnk[i].state, SMC_LINK_DOWN);
+	}
+out:
+	llc_kill_waiters(lgr);
+	cancel_work_sync(&lgr->llc_event_work);
+	schedule_delayed_work(&lgr->free_lgr_final_work, 2 * TCP_TIMEWAIT_LEN);
+}
+
+static void smc_free_lgr_final_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  free_lgr_final_work.work);
+
+	smc_free_lgr(lgr);
+}
+
+static inline int smc_splice_in_use(const struct smc_sock *smc)
+{
+	return smc->spd != NULL;
+}
+
+static int smc_splice_in_progress(struct smc_sock *smc)
+{
+	struct smc_curs curs;
+	struct smc_curs spl_curs;
+
+	spl_curs.s.lcurs = atomic64_read(&smc->conn.local_rx_spliced.s.acurs);
+	if (spl_curs.s.curs.reserved0) {
+		curs.s.lcurs =
+		       atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+		if (smc_curs_diff(smc->conn.rmb_rx_size, &curs, &spl_curs))
+			return 1;
+	}
+
+	return 0;
+}
+
+static inline int smc_find_pending_send(struct smc_link *link, u64 wr_id)
+{
+	u32 i;
+
+	for (i = 0; i < link->send_wr_num; i++) {
+		if (link->pending[i].wr_id_send == wr_id)
+			return i;
+	}
+	return link->send_wr_num;
+}
+
+static void smc_flag_sock_err(struct smc_sock *smc)
+{
+	int old_state;
+
+	sock_hold(&smc->sk);
+	smc->sk.sk_err = ECONNABORTED;
+	lock_sock(&smc->sk);
+	old_state = smc->sk.sk_state;
+	smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+	if (smc->tcpsocket && smc->tcpsocket->sk) {
+		smc->tcpsocket->sk->sk_err = ECONNABORTED;
+		smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+	}
+	smc_conn_term(smc);
+	if (!sock_flag(&smc->sk, SOCK_DEAD))
+		smc->sk.sk_error_report(&smc->sk);
+	release_sock(&smc->sk);
+	if (old_state != SMC_INIT) {
+		/* forget about data prepared to be sent */
+		atomic64_set(&smc->conn.tx_curs_prep.s.acurs,
+			     atomic64_read(&smc->conn.tx_curs_sent.s.acurs));
+		wake_up_interruptible(&smc->rx_waiter);
+		smc_sock_wake_tx(&smc->sk);
+		smc_sock_wake_rx(&smc->sk);
+		wake_up(&smc->prep_waiter);
+	}
+	sock_put(&smc->sk);
+	if ((old_state != SMC_INIT) &&
+	    (smc->sk.sk_state == SMC_CLOSED) &&
+	    (sock_flag(&smc->sk, SOCK_DEAD)) &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		schedule_work(&smc->release_work);
+	}
+}
+
+void smc_terminate_conn(struct smc_link_group *lgr)
+{
+	struct smc_sock *smc;
+	struct rb_node *node;
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	if (list_empty(&lgr->list)) { /* already terminated ? */
+		spin_unlock_bh(&smc_lgr_list.lock);
+		return;
+	}
+	list_del_init(&lgr->list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+again:
+	read_lock_bh(&lgr->conns_lock);
+	for (node = rb_first(&lgr->conns_all); node; node = rb_next(node)) {
+		smc = container_of(rb_entry(node, struct smc_connection,
+					    alert_node),
+				   struct smc_sock, conn);
+		if ((smc->sk.sk_err != ECONNABORTED) && (!smc->use_tcp)) {
+			read_unlock_bh(&lgr->conns_lock);
+			smc_flag_sock_err(smc);
+			goto again;
+		}
+	}
+	read_unlock_bh(&lgr->conns_lock);
+}
+
+static int smc_write_space(struct smc_sock *);
+
+static int smc_switch_cursor(struct smc_sock *smc, int to_lnk_idx)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_curs cons, fin;
+	int diff;
+	int rc = 0;
+
+	atomic64_set(&conn->tx_curs_sent.s.acurs,
+		     atomic64_read(&conn->tx_curs_fin.s.acurs));
+	fin.s.lcurs = atomic64_read(&conn->local_tx_ctrl_fin.s.acurs);
+	atomic64_set(&conn->local_tx_ctrl.p_curs.s.acurs, fin.s.lcurs);
+	cons.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	if (((cons.s.curs.w > fin.s.curs.w) ||
+	     ((cons.s.curs.w == fin.s.curs.w) &&
+	      (cons.s.curs.c > fin.s.curs.c))) ||
+	    (!cons.s.curs.w && (fin.s.curs.w == 0xffff))) {
+		diff = smc_curs_diff(conn->peer_rx_buf_len, &fin, &cons);
+		smc_curs_add(conn->rmb_tx_size,
+			     &conn->tx_curs_sent.s.curs, diff);
+		smc_curs_add(conn->rmb_tx_size,
+			     &conn->tx_curs_fin.s.curs, diff);
+		smp_mb__before_atomic();
+		atomic_add(diff, &conn->tx_buf_space);
+		smp_mb__after_atomic();
+		smc_curs_add(conn->peer_rx_buf_len,
+			     &conn->local_tx_ctrl.p_curs.s.curs, diff);
+		smc_curs_add(conn->peer_rx_buf_len,
+			     &conn->local_tx_ctrl_fin.s.curs, diff);
+	}
+	atomic_set(&smc->conn.rx_buf_space, smc_write_space(smc));
+	if ((smc->sk.sk_state != SMC_INIT) &&
+	    (smc->sk.sk_state != SMC_CLOSED) &&
+	    (smc->sk.sk_state != SMC_DESTRUCT)) {
+		conn->local_tx_ctrl.p_flags.last_msg_valid_req = 1;
+		rc = smc_ctrl_send(conn, conn->lnk_idx);
+		conn->local_tx_ctrl.p_flags.last_msg_valid_req = 0;
+		schedule_delayed_work(&smc->write_work, 0);
+		smc_sock_wake_rx_for_tx(&smc->sk);
+	}
+	return rc;
+}
+
+int smc_switch_conns(struct smc_link_group *lgr, int lnk_idx, int dev_err)
+{
+	struct smc_sock *smc;
+	struct smc_connection *conn;
+	struct smc_roce_defs *rd_idx = &lgr->lnk[lnk_idx].roce_defs;
+	int to_lnk_idx, i, rc = 0, found = 0;
+	struct rb_node *node;
+
+	wake_up(&lgr->lnk[lnk_idx].wqe_waiter);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		struct smc_roce_defs *rd_i = &lgr->lnk[i].roce_defs;
+
+		if ((atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP) ||
+		    (i == lnk_idx))
+			continue;
+		if (dev_err && (rd_i->ibdev == rd_idx->ibdev) &&
+		    (rd_i->port == rd_idx->port))
+			continue;
+		found = 1;
+		to_lnk_idx = i;
+		break;
+	}
+	if (!found) {
+		lgr->lgr_type = NONE;
+		wake_up_interruptible(&lgr->llc_waiter);
+		smc_terminate_conn(lgr); /* requires lgr_list_lock and
+					  * lgr->conns_lock
+					  */
+		return -ENOENT;
+	}
+
+again:
+	read_lock_bh(&lgr->conns_lock);
+	for (node = rb_first(&lgr->conns_all); node; node = rb_next(node)) {
+		conn = rb_entry(node, struct smc_connection, alert_node);
+		if (conn->lnk_idx != lnk_idx)
+			continue;
+		smc = container_of(conn, struct smc_sock, conn);
+		if ((smc->sk.sk_state == SMC_CLOSED) ||
+		    (smc->sk.sk_state == SMC_PEERCLW2) ||
+		    (smc->sk.sk_state == SMC_PEERFINCLW) ||
+		    (smc->sk.sk_state == SMC_APPLFINCLW) ||
+		    (smc->sk.sk_state == SMC_APPLCLW2) ||
+		    (smc->sk.sk_state == SMC_PROCESSABORT)) {
+			conn->lnk_idx = to_lnk_idx;
+			continue;
+		}
+		sock_hold(&smc->sk);
+		read_unlock_bh(&lgr->conns_lock);
+		lock_sock(&smc->sk);
+		conn->lnk_idx = to_lnk_idx;
+		if (!(atomic64_read(&conn->tx_curs_prep.s.acurs)) &&
+		    !(atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs)) &&
+		    !conn->local_tx_ctrl.conn_state_flags.closed_conn) {
+			release_sock(&smc->sk);
+			sock_put(&smc->sk);
+			goto again;
+		}
+		rc = smc_switch_cursor(smc, to_lnk_idx);
+		release_sock(&smc->sk);
+		sock_put(&smc->sk);
+		if (rc) {
+			smc_link_down(lgr, to_lnk_idx);
+			break;
+		}
+		goto again;
+	}
+	read_unlock_bh(&lgr->conns_lock);
+	return to_lnk_idx;
+}
+
+void smc_cq_event_handler(struct ib_event *evt, void *ptr)
+{
+	return;
+}
+
+static void smc_qp_event_handler(struct ib_event *evt, void *ptr)
+{
+	struct smc_link *link = (struct smc_link *)ptr;
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_roce_defs *rocdefs = &link->roce_defs;
+	struct smc_ib_device *smc_ibdev = (struct smc_ib_device *)evt->device;
+	int lnk_idx;
+
+	lnk_idx = link - &lgr->lnk[0];
+
+	switch (evt->event) {
+	case IB_EVENT_DEVICE_FATAL:	/* 8 */
+		if (cmpxchg(&smc_ibdev->port_err[rocdefs->port - 1], 0, 1) == 0)
+			schedule_work(&rocdefs->ibdev->port_err_work);
+		break;
+	case IB_EVENT_GID_CHANGE:	 /* 17 */
+	case IB_EVENT_PORT_ERR:		 /* 10 */
+	case IB_EVENT_QP_ACCESS_ERR:	 /* 3 */
+		if ((atomic_read(&link->state) == SMC_LINK_UP) &&
+		    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0))
+			schedule_work(&lgr->link_down_work);
+		break;
+	case IB_EVENT_COMM_EST:		 /* 4 */
+	case IB_EVENT_PORT_ACTIVE:	 /* 9 */
+	default:
+		break;
+	}
+}
+
+static inline void smc_send_tasklet_conn(struct smc_sock *smc)
+{
+	if (smc->sk.sk_socket && atomic_read(&smc->conn.tx_buf_space) &&
+	    test_bit(SOCK_NOSPACE, &smc->sk.sk_socket->flags)) {
+		smc_sock_wake_tx(&smc->sk);
+	}
+	if (smc->sk.sk_state != SMC_ACTIVE)
+		wake_up(&smc->destruct_waiter);
+}
+
+static inline void smc_send_tasklet_wc(struct ib_wc *wc)
+{
+	struct smc_link_group *lgr;
+	struct smc_link *link;
+	struct smc_sock *smc;
+	struct pending_send pnd_snd;
+	u32 pnd_snd_idx;
+	int i, diff;
+
+	link = wc->qp->qp_context;
+	if (!link->link_id ||
+	    (memcmp(link->lnk_eyecatch, SMC_LNK_EYE_CATCHER,
+		    SMC_LNK_EYE_CATCH_LEN)))
+		return;
+	pnd_snd_idx = smc_find_pending_send(link, wc->wr_id);
+	if (pnd_snd_idx == link->send_wr_num)
+		return;
+	memcpy(&pnd_snd, &link->pending[pnd_snd_idx],
+	       sizeof(struct pending_send));
+	if (!test_and_clear_bit(pnd_snd_idx, link->pending_bit_mask))
+		return;
+	lgr = link->lnk_grp;
+	if (wc->status) {
+		int lnk_idx = link - &lgr->lnk[0];
+
+		for_each_set_bit(i, link->pending_bit_mask, link->send_wr_num)
+			clear_bit(i, link->pending_bit_mask);
+		if ((wc->wr_id != 1) &&
+		    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0))
+			schedule_work(&lgr->link_down_work);
+	}
+	if (pnd_snd.conn) {
+		smc = container_of(pnd_snd.conn, struct smc_sock, conn);
+		bh_lock_sock(&smc->sk);
+		if ((!wc->status) &&
+		    (link == &lgr->lnk[pnd_snd.conn->lnk_idx])) {
+			diff = smc_curs_diff(pnd_snd.conn->rmb_tx_size,
+					     &pnd_snd.conn->tx_curs_fin,
+					     &pnd_snd.cursor);
+			smp_mb__before_atomic();
+			atomic_add(diff, &pnd_snd.conn->tx_buf_space);
+			smp_mb__after_atomic();
+			atomic64_set(&pnd_snd.conn->tx_curs_fin.s.acurs,
+				     pnd_snd.cursor.s.lcurs);
+			atomic64_set(&pnd_snd.conn->local_tx_ctrl_fin.s.acurs,
+				     pnd_snd.p_cursor.s.lcurs);
+			pnd_snd.conn->tx_seq_fin = pnd_snd.ctrl_seq;
+		}
+		wake_up(&link->wqe_waiter);
+		smc_send_tasklet_conn(smc);
+		bh_unlock_sock(&smc->sk);
+	} else {
+		if (pnd_snd.post_ctl != LLC_NO_NOTIFY) {
+			lgr->llc_ctl[pnd_snd.post_ctl].ctrl_elem =
+							LLC_SEND_POSTED;
+			lgr->llc_ctl[pnd_snd.post_ctl].wc_status = wc->status;
+			wake_up_interruptible(&lgr->llc_waiter);
+		}
+		wake_up(&link->wqe_waiter);
+	}
+}
+
+void smc_send_tasklet_fn(unsigned long data)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)data;
+	struct ib_wc wc[SMC_MAX_POLL_CQE];
+	int i = 0, rc, rc1 = 0;
+	int polled = 0;
+
+again:
+	polled++;
+	do {
+		rc = ib_poll_cq(ibdev->roce_cq_send, SMC_MAX_POLL_CQE, wc);
+		if (polled == 1) {
+			rc1 = ib_req_notify_cq(ibdev->roce_cq_send,
+					       IB_CQ_NEXT_COMP |
+					       IB_CQ_REPORT_MISSED_EVENTS);
+		}
+		if (!rc)
+			break;
+		for (i = 0; i < rc; i++)
+			smc_send_tasklet_wc(&wc[i]);
+	} while (rc > 0);
+	if (polled == 1)
+		goto again;
+}
+
+void smc_cq_handler_send(struct ib_cq *ib_cq, void *cq_context)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)cq_context;
+
+	tasklet_schedule(&ibdev->send_tasklet);
+}
+
+static inline
+void smc_save_cursor(struct smc_curs *local, struct rmb_curs *peer)
+{
+	struct smc_curs temp, old;
+
+	old.s.lcurs = atomic64_read(&local->s.acurs);
+	temp.s.lcurs = atomic64_read(&peer->r.acurs);
+	temp.s.curs.c = ntohl(temp.s.curs.c);
+	temp.s.curs.w = ntohs(temp.s.curs.w);
+	if ((old.s.curs.w > temp.s.curs.w) && temp.s.curs.w)
+		return;
+	if ((old.s.curs.w == temp.s.curs.w) &&
+	    (old.s.curs.c > temp.s.curs.c))
+		return;
+	atomic64_set(&local->s.acurs, temp.s.lcurs);
+}
+
+static inline
+void smc_set_cursor(struct smc_curs *local, struct rmb_cursor *peer)
+{
+	struct smc_curs temp;
+
+	temp.s.lcurs = atomic64_read(&local->s.acurs);
+	peer->count = htonl(temp.s.curs.c);
+	peer->wwrap_seq = htons(temp.s.curs.w);
+}
+
+static inline
+void smc_save_ctrl(struct smc_e_ctrl *local, struct rmb_e_ctrl *peer)
+{
+	local->p_flags = peer->prod_flags;
+	local->conn_state_flags = peer->conn_state_flags;
+	smc_save_cursor(&local->p_curs, &peer->p_curs);
+	smc_save_cursor(&local->c_curs, &peer->c_curs);
+	local->ctrl_type = peer->ctrl_type;
+	local->ctrl_len = peer->ctrl_len;
+	local->ctrl_seq = ntohs(peer->ctrl_seq);
+	local->ctrl_token = peer->ctrl_token;
+}
+
+static inline
+void smc_set_ctrl(struct smc_e_ctrl *local, struct rmb_e_ctrl *peer)
+{
+	memcpy((char *)peer, (char *)local, SMC_SIZE_OF_CTRL_DATA);
+	peer->prod_flags = local->p_flags;
+	peer->conn_state_flags = local->conn_state_flags;
+	smc_set_cursor(&local->p_curs, &peer->p_curs.r.curs);
+	smc_set_cursor(&local->c_curs, &peer->c_curs.r.curs);
+	peer->ctrl_seq = htons(local->ctrl_seq);
+	peer->ctrl_token = local->ctrl_token;
+}
+
+static void smc_update_cons_curs(struct smc_sock *, struct smc_link *link);
+
+static int smc_recv_validate(struct smc_sock *smc, struct smc_link *link,
+			     u32 index)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+	u16 tmp;
+
+	/* check that seqnum was seen before */
+	tmp = conn->ctrl_seq_rcvd - ntohs(link->ctrl_buf_rx[index].ctrl_seq);
+	if (tmp > 0x7fff) {
+		/* drop connection */
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+		conn->local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			smc->tcpsocket->sk->sk_err = ECONNABORTED;
+			smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+		}
+		conn->lnk_idx = link - &conn->lgr->lnk[0];
+		smc_conn_term(smc);
+		wake_up_interruptible(&smc->rx_waiter);
+		smc_sock_wake_tx(&smc->sk);
+		smc_sock_wake_rx(&smc->sk);
+		if ((smc->sk.sk_state == SMC_CLOSED) &&
+		    (sock_flag(&smc->sk, SOCK_DEAD)) &&
+		    (cmpxchg(&smc->sock_put_done, 0, 1) == 0))
+			schedule_work(&smc->release_work);
+		rc = -ECONNABORTED;
+	} else {
+		conn->local_rx_ctrl.p_flags.cons_curs_upd_req = 1;
+	}
+	return rc;
+}
+
+static void smc_recv_action(struct smc_sock *smc,
+			    struct smc_link *link, u32 index)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_curs prod, cons_old, prod_old;
+	int diff;
+
+	if (link->ctrl_buf_rx[index].prod_flags.last_msg_valid_req) {
+		if (smc_recv_validate(smc, link, index))
+			return;
+	}
+	prod_old.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons_old.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	smc_save_ctrl(&conn->local_rx_ctrl, &link->ctrl_buf_rx[index]);
+	if (!conn->local_rx_ctrl.p_flags.last_msg_valid_req) {
+		if ((conn->local_rx_ctrl.ctrl_seq -
+			(u16)(conn->ctrl_seq_rcvd + 1)) > 0x7fff)
+			return;
+		conn->ctrl_seq_rcvd = conn->local_rx_ctrl.ctrl_seq;
+	}
+	diff = smc_curs_diff(conn->peer_rx_buf_len, &cons_old,
+			     &conn->local_rx_ctrl.c_curs);
+	smp_mb__before_atomic();
+	atomic_add(diff, &conn->rx_buf_space);
+	smp_mb__after_atomic();
+	diff = smc_curs_diff(conn->rmb_rx_size, &prod_old,
+			     &conn->local_rx_ctrl.p_curs);
+	smp_mb__before_atomic();
+	atomic_add(diff, &conn->bytes_to_rcv);
+	smp_mb__after_atomic();
+
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		smc->sk.sk_err = ECONNRESET;
+	if (smc_stop_received(conn)) {
+		smc->sk.sk_shutdown |= RCV_SHUTDOWN;
+		sock_set_flag(&smc->sk, SOCK_DONE);
+		smc_conn_term(smc);
+		smc_sock_wake_rx(&smc->sk);
+		smc_sock_wake_tx(&smc->sk);
+	}
+
+	/* piggy backed tx info */
+	smc_sock_wake_rx_for_tx(&smc->sk);
+
+	if ((smc->sk.sk_state == SMC_CLOSED) &&
+	    smc_close_received(conn) &&
+	    sock_flag(&smc->sk, SOCK_DEAD) &&
+	    !smc->sk.sk_socket &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		wake_up_interruptible(&smc->rx_waiter);
+		/* socket is closed and peer has confirmed close, now we are
+		 * allowed to clean up our sock/rmbe
+		 */
+		schedule_work(&smc->release_work);
+		return;
+	}
+
+	/* socket connected but not accepted */
+	if (!smc->sk.sk_socket)
+		return;
+
+	/* data available */
+	if (conn->local_rx_ctrl.p_flags.urg_data_present) {
+		prod.s.lcurs =
+			atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+		smc_curs_dec(conn->rmb_rx_size, &prod.s.curs);
+		atomic64_set(&conn->rx_urg_curs.s.acurs, prod.s.lcurs);
+		/* set urg_data to previous byte */
+		conn->rx_urg_data =
+			conn->rmb_rx_elem->rmb_rx[prod.s.curs.c];
+		conn->rx_urg_state = SMC_URG_VALID;
+		sk_send_sigurg(&smc->sk);
+	}
+	if ((conn->local_rx_ctrl.p_flags.write_blocked) ||
+	    (conn->local_rx_ctrl.p_flags.cons_curs_upd_req))
+		smc_update_cons_curs(smc, link);
+	if ((smc_to_read(smc) > 0) ||
+	    conn->local_rx_ctrl.p_flags.urg_data_pending ||
+	    smc_stop_received(conn) ||
+	    smc->sk.sk_shutdown & RCV_SHUTDOWN)
+		smc_sock_wake_rx(&smc->sk);
+}
+
+static inline void smc_recv_conn_update(struct rmb_e_ctrl *ctrl,
+					struct smc_link *link, u64 wr_id)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_connection *connection;
+	struct smc_sock *smc;
+	u32 index;
+
+	/* lookup connection */
+	read_lock_bh(&lgr->conns_lock);
+	connection = smc_find_conn(ctrl->ctrl_token, lgr);
+	if (!connection) {
+		read_unlock_bh(&lgr->conns_lock);
+	} else {
+		smc = container_of(connection, struct smc_sock, conn);
+		if (smc->sk.sk_state != SMC_DESTRUCT) {
+			sock_hold(&smc->sk);
+			read_unlock_bh(&lgr->conns_lock);
+			index = wr_id % link->recv_wr_num;
+			bh_lock_sock(&smc->sk);
+			smc_recv_action(smc, link, index);
+			bh_unlock_sock(&smc->sk);
+			sock_put(&smc->sk); /* no free sk in softirq-context */
+		} else {
+			read_unlock_bh(&lgr->conns_lock);
+		}
+	}
+}
+
+static inline void smc_recv_process_wqs(struct ib_wc wc[], int num)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct rmb_e_ctrl *ctrl;
+	int i, idx;
+	u32 index;
+
+	for (i = 0; i < num; i++) {
+		link = wc[i].qp->qp_context;
+		if (memcmp(link->lnk_eyecatch, SMC_LNK_EYE_CATCHER,
+			   SMC_LNK_EYE_CATCH_LEN))
+			continue;
+		lgr = link->lnk_grp;
+		if ((atomic_read(&link->state) == SMC_LINK_DOWN) ||
+		    (atomic_read(&link->state) == SMC_LINK_FREED))
+			continue;
+		if (wc[i].status == IB_WC_SUCCESS) {
+			index = wc[i].wr_id % link->recv_wr_num;
+			ctrl = (struct rmb_e_ctrl *)&link->ctrl_buf_rx[index];
+			link->rcv_tstamp = (u32)jiffies;
+			if (ctrl->ctrl_type == RMBE_CTRL)
+				smc_recv_conn_update(ctrl, link, wc[i].wr_id);
+			else
+				llc_enqueue(link, (struct llc_msg *)ctrl);
+			smc_outstanding_post_recv(link);
+		} else {
+			/* handle status errors
+			 * terminate all connections of this lgr abnormally
+			 */
+			switch (wc[i].status) {
+			case IB_WC_RETRY_EXC_ERR:
+			case IB_WC_RNR_RETRY_EXC_ERR:
+			case IB_WC_WR_FLUSH_ERR:
+				idx = link - &lgr->lnk[0];
+				if ((atomic_read(&link->state) == SMC_LINK_UP)
+				 && (cmpxchg(&lgr->lnk_down[idx], 0, 1) == 0))
+					schedule_work(&lgr->link_down_work);
+				break;
+			default:
+				smc_outstanding_post_recv(link);
+				break;
+			}
+			continue;
+		}
+	}
+}
+
+void smc_recv_tasklet_fn(unsigned long data)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)data;
+	struct ib_wc wc[SMC_MAX_POLL_CQE];
+	int rc;
+	int polled = 0;
+
+again:
+	polled++;
+	do {
+		memset(&wc, 0, sizeof(wc));
+		rc = ib_poll_cq(ibdev->roce_cq_recv, SMC_MAX_POLL_CQE, wc);
+		if (polled == 1)
+			ib_req_notify_cq(ibdev->roce_cq_recv,
+					 IB_CQ_SOLICITED_MASK
+				       | IB_CQ_REPORT_MISSED_EVENTS);
+		if (!rc)
+			break;
+		smc_recv_process_wqs(&wc[0], rc);
+	} while (rc > 0);
+	if (polled == 1)
+		goto again;
+}
+
+void smc_cq_handler_recv(struct ib_cq *ib_cq, void *cq_context)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)cq_context;
+
+	tasklet_schedule(&ibdev->recv_tasklet);
+}
+
+static int smc_alloc_link_mem(struct smc_link *link)
+{
+	/* allocate link related memory */
+	link->ctrl_buf_rx = (struct rmb_e_ctrl *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct rmb_e_ctrl),
+			GFP_KERNEL);
+	if (!link->ctrl_buf_rx)
+		goto no_mem;
+	link->ctrl_buf_tx = (struct rmb_e_ctrl *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct rmb_e_ctrl),
+			GFP_KERNEL);
+	if (!link->ctrl_buf_tx)
+		goto no_mem_ctrl_buf_rx;
+	link->send_wr = (struct ib_send_wr *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct ib_send_wr),
+			GFP_KERNEL);
+	if (!link->send_wr)
+		goto no_mem_ctrl_buf_tx;
+	link->recv_wr = (struct ib_recv_wr *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct ib_recv_wr),
+			GFP_KERNEL);
+	if (!link->recv_wr)
+		goto no_mem_send_wr;
+	link->send_sge = (struct ib_sge *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct ib_sge),
+			GFP_KERNEL);
+	if (!link->send_sge)
+		goto no_mem_recv_wr;
+	link->recv_sge = (struct ib_sge *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct ib_sge),
+			GFP_KERNEL);
+	if (!link->recv_sge)
+		goto no_mem_send_sge;
+	link->pending_bit_mask = kzalloc(BITS_TO_LONGS(smc_ctrl_buf_cnt) *
+					 sizeof(unsigned long),
+					 GFP_KERNEL);
+	if (!link->pending_bit_mask)
+		goto no_mem_recv_sge;
+	link->pending = (struct pending_send *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct pending_send),
+			GFP_KERNEL);
+	if (!link->pending)
+		goto no_mem_pending_bit_mask;
+	return 0;
+
+no_mem_pending_bit_mask:
+	kfree(link->pending_bit_mask);
+no_mem_recv_sge:
+	kfree(link->recv_sge);
+no_mem_send_sge:
+	kfree(link->send_sge);
+no_mem_recv_wr:
+	kfree(link->recv_wr);
+no_mem_send_wr:
+	kfree(link->send_wr);
+no_mem_ctrl_buf_tx:
+	kfree(link->ctrl_buf_tx);
+no_mem_ctrl_buf_rx:
+	kfree(link->ctrl_buf_rx);
+no_mem:
+	return -ENOMEM;
+}
+
+static void smc_init_sge(struct smc_link *lnk)
+{
+	u32 i;
+
+	for (i = 0; i < lnk->send_wr_num; i++) {
+		lnk->send_wr[i].next = NULL;
+		lnk->send_wr[i].sg_list = &lnk->send_sge[i];
+		lnk->send_wr[i].num_sge = 1;
+		lnk->send_wr[i].opcode = IB_WR_SEND;
+		lnk->send_wr[i].send_flags = IB_SEND_SIGNALED
+					   | IB_SEND_SOLICITED | IB_SEND_INLINE;
+		lnk->send_sge[i].addr =
+				lnk->ctrl_dma_tx + i * SMC_SIZE_OF_CTRL_BUF;
+		lnk->send_sge[i].length = SMC_SIZE_OF_CTRL_DATA;
+		lnk->send_sge[i].lkey = lnk->mr_tx->lkey;
+	}
+	for (i = 0; i < lnk->recv_wr_num; i++) {
+		lnk->recv_wr[i].next = NULL;
+		lnk->recv_wr[i].sg_list = &lnk->recv_sge[i];
+		lnk->recv_wr[i].num_sge = 1;
+		lnk->recv_sge[i].addr =
+				lnk->ctrl_dma_rx + i * SMC_SIZE_OF_CTRL_BUF;
+		lnk->recv_sge[i].length = SMC_SIZE_OF_CTRL_BUF;
+		lnk->recv_sge[i].lkey = lnk->mr_tx->lkey;
+	}
+}
+
+int smc_create_link(struct smc_link_group *lgr, int lnk_idx,
+		    struct smc_roce_defs *rocdefs)
+{
+	struct smc_ib_device *smc_ibdev = rocdefs->ibdev;
+	struct ib_device *ibdev;
+	struct smc_link *lnk = &lgr->lnk[lnk_idx];
+	int i, rc = 0;
+
+	if (list_empty(&smc_ib_devices))
+		return -ENODEV;
+	atomic_set(&lnk->state, SMC_LINK_ACTIVATING);
+	init_waitqueue_head(&lnk->wqe_waiter);
+	ibdev = smc_ibdev->dev;
+	memcpy(&lnk->roce_defs, rocdefs, sizeof(struct smc_roce_defs));
+	lnk->path_mtu = smc_ibdev->pattr[rocdefs->port - 1].active_mtu;
+	while (1) {
+		lnk->link_id = atomic_inc_return(&lgr->link_num);
+		if (!lnk->link_id)
+			continue;
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			if ((i != lnk_idx) &&
+			    (lgr->lnk[i].link_id == lnk->link_id))
+				continue;
+		}
+		break;
+	}
+	/* create PD */
+	lnk->roce_pd = ib_alloc_pd(ibdev);
+	if (IS_ERR(lnk->roce_pd)) {
+		atomic_set(&lnk->state, SMC_LINK_FREED);
+		rc = PTR_ERR(lnk->roce_pd);
+		lnk->roce_pd = NULL;
+		return rc;
+	}
+	/* create QP */
+	{
+	struct ib_qp_init_attr qp_attr = {
+		.event_handler = smc_qp_event_handler,
+		.qp_context = lnk,
+		.send_cq = smc_ibdev->roce_cq_send,
+		.recv_cq = smc_ibdev->roce_cq_recv,
+		.srq = NULL,
+		.cap = {
+			.max_send_wr = 3 * smc_ctrl_buf_cnt,
+				/* include unsolicited rdma_writes as well,
+				 * otherwise mlx4 code reaches overflow
+				 */
+			.max_recv_wr = 10 * smc_ctrl_buf_cnt,
+			.max_send_sge = 4,
+			.max_recv_sge = 1,
+			.max_inline_data = SMC_SIZE_OF_CTRL_BUF,
+		},
+		.sq_sig_type = IB_SIGNAL_REQ_WR,
+		.qp_type = IB_QPT_RC,
+	};
+	lnk->roce_qp = ib_create_qp(lnk->roce_pd, &qp_attr);
+	if (IS_ERR(lnk->roce_qp)) {
+		rc = PTR_ERR(lnk->roce_qp);
+		lnk->roce_qp = NULL;
+		goto dealloc_pd;
+	}
+	}
+	smc_check_qp_attr(lnk);
+	lnk->wr_id_send = 0;
+	lnk->wr_id_recv = 0;
+	lnk->mr_tx = ib_get_dma_mr(lnk->roce_pd, IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(lnk->mr_tx)) {
+		rc = PTR_ERR(lnk->mr_tx);
+		lnk->mr_tx = NULL;
+		goto destroy_qp;
+	}
+	lnk->ctrl_dma_rx = ib_dma_map_single(ibdev, lnk->ctrl_buf_rx,
+				(sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+					     DMA_FROM_DEVICE);
+	if (ib_dma_mapping_error(ibdev, lnk->ctrl_dma_rx)) {
+		lnk->ctrl_dma_rx = 0;
+		rc = -EIO;
+		goto dereg_mr;
+	}
+	lnk->ctrl_dma_tx = ib_dma_map_single(ibdev, lnk->ctrl_buf_tx,
+				sizeof(struct rmb_e_ctrl) * lnk->send_wr_num,
+					     DMA_TO_DEVICE);
+	if (ib_dma_mapping_error(ibdev, lnk->ctrl_dma_tx)) {
+		rc = -EIO;
+		goto dma_unmap;
+	}
+	get_random_bytes_arch(((char *)&lnk->psn_initial) + 1, 3);
+	lnk->rcv_tstamp = 0;
+	lnk->probes_out = 0;
+	smc_init_sge(lnk);
+	memcpy(lnk->lnk_eyecatch, SMC_LNK_EYE_CATCHER, SMC_LNK_EYE_CATCH_LEN);
+	memset(lnk->pending_bit_mask, 0,
+	       BITS_TO_LONGS(smc_ctrl_buf_cnt) * sizeof(unsigned long));
+	return 0;
+
+dma_unmap:
+	ib_dma_unmap_single(ibdev, lnk->ctrl_dma_rx,
+			    (sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+			    DMA_FROM_DEVICE);
+	lnk->ctrl_dma_rx = 0;
+dereg_mr:
+	ib_dereg_mr(lnk->mr_tx);
+	lnk->mr_tx = NULL;
+destroy_qp:
+	ib_destroy_qp(lnk->roce_qp);
+	lnk->roce_qp = NULL;
+dealloc_pd:
+	ib_dealloc_pd(lnk->roce_pd);
+	lnk->roce_pd = NULL;
+	lnk->link_id = 0;
+	memset(&lnk->roce_defs, 0, sizeof(struct smc_roce_defs));
+	atomic_set(&lnk->state, SMC_LINK_FREED);
+	return rc;
+}
+
+static inline
+struct rmb_tx_addrs *smc_get_tx_rmb_slot(struct smc_link_group *lgr,
+					 int start)
+{
+	struct rmb_tx_addrs *slot;
+
+	list_for_each_entry(slot, &lgr->rmb_tx_bufs[start], list) {
+		if (cmpxchg(&slot->used, 0, 1) == 0)
+			return slot;
+	}
+	return NULL;
+}
+
+static inline
+struct rmb_rx_addrs *smc_get_rx_rmb_slot(struct smc_link_group *lgr,
+					 int start)
+{
+	struct rmb_rx_addrs *slot;
+
+	list_for_each_entry(slot, &lgr->rmb_rx_bufs[start], list) {
+		if (cmpxchg(&slot->used, 0, 1) == 0)
+			return slot;
+	}
+	return NULL;
+}
+
+int smc_map_rmbs_tx(struct smc_link_group *lgr, int lnk_idx, int tx_size,
+		    struct rmb_tx_addrs *elem)
+{
+	struct smc_ib_device *dev = lgr->lnk[lnk_idx].roce_defs.ibdev;
+	int rc = 0;
+
+	if (elem->rmb_tx_dma[lnk_idx])
+		return rc; /* already mapped */
+	elem->rmb_tx_dma[lnk_idx] =
+		(char *)ib_dma_map_single(dev->dev, elem->rmb_tx,
+					  tx_size, DMA_TO_DEVICE);
+	if (ib_dma_mapping_error(dev->dev,
+				 (dma_addr_t)elem->rmb_tx_dma[lnk_idx]))
+		rc = -EIO;
+	return rc;
+}
+
+int smc_map_rmbs_rx(struct smc_link_group *lgr, int lnk_idx, int rx_size,
+		    struct rmb_rx_addrs *elem)
+{
+	struct smc_ib_device *dev = lgr->lnk[lnk_idx].roce_defs.ibdev;
+	int rc = 0;
+
+	if (elem->rmb_rx_dma[lnk_idx])
+		return rc; /* already mapped */
+	elem->rmb_rx_dma[lnk_idx] =
+		(char *)ib_dma_map_single(dev->dev, elem->rmb_rx,
+					  rx_size, DMA_FROM_DEVICE);
+	if (ib_dma_mapping_error(dev->dev,
+				 (dma_addr_t)elem->rmb_rx_dma[lnk_idx]))
+		rc = -EIO;
+	return rc;
+}
+
+long smc_get_dma_mr(struct smc_link_group *lgr, int lnk_idx,
+		    struct rmb_rx_addrs *elem)
+{
+	struct ib_mr *mr;
+	long rc = 0;
+
+	if (elem->mr_rx[lnk_idx])
+		return 0; /* already done */
+	/* obtain unique key -
+	 * next invocation of ib_get_dma_mr returns a different key!
+	 */
+	mr = ib_get_dma_mr(lgr->lnk[lnk_idx].roce_pd,
+			   IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE);
+	if (!IS_ERR(mr)) {
+		elem->mr_rx[lnk_idx] = mr;
+	} else {
+		elem->mr_rx[lnk_idx] = NULL;
+		rc = PTR_ERR(mr);
+	}
+	return rc;
+}
+
+long smc_map_rmbs_to_link(struct smc_link_group *lgr, int lnk_idx,
+			  int reuse_lnk_idx)
+{
+	struct rmb_tx_addrs *tx_entry;
+	struct rmb_rx_addrs *rx_entry;
+	long rc = 0;
+	int i;
+
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(tx_entry, &lgr->rmb_tx_bufs[i], list) {
+			if (reuse_lnk_idx <= SMC_MAX_SYM_LINKS) {
+				tx_entry->rmb_tx_dma[lnk_idx] =
+					tx_entry->rmb_tx_dma[reuse_lnk_idx];
+			} else {
+				rc = smc_map_rmbs_tx(lgr, lnk_idx,
+					     smc_uncompress_bufsize(rol8(i, 4)),
+						     tx_entry);
+				if (rc)
+					goto out;
+			}
+		}
+		list_for_each_entry(rx_entry, &lgr->rmb_rx_bufs[i], list) {
+			if (reuse_lnk_idx <= SMC_MAX_SYM_LINKS) {
+				rx_entry->rmb_rx_dma[lnk_idx] =
+					rx_entry->rmb_rx_dma[reuse_lnk_idx];
+			} else {
+				rc = smc_map_rmbs_rx(lgr, lnk_idx,
+					     smc_uncompress_bufsize(rol8(i, 4)),
+						     rx_entry);
+				if (rc)
+					goto out;
+			}
+			rc = smc_get_dma_mr(lgr, lnk_idx, rx_entry);
+			if (rc)
+				goto out;
+		}
+	}
+	return rc;
+
+out:
+	smc_link_down(lgr, lnk_idx);
+	return rc;
+}
+
+long smc_map_rmbs(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	long rc = 0;
+	int i, j;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP)
+			continue;
+		for (j = 0; j < i; j++) {
+			if ((lgr->lnk[i].roce_defs.ibdev ==
+				lgr->lnk[j].roce_defs.ibdev) &&
+			    (lgr->lnk[i].roce_defs.port ==
+				lgr->lnk[j].roce_defs.port))
+				break;
+		}
+		if (!smc->conn.tx_rmb_reused) {
+			if (j == i) {
+				rc = smc_map_rmbs_tx(lgr, i,
+						     smc->conn.rmb_tx_size,
+						     smc->conn.rmb_tx_elem);
+				if (rc) {
+					smc_link_down(lgr, i);
+					continue;
+				}
+			} else {
+				smc->conn.rmb_tx_elem->rmb_tx_dma[i] =
+					smc->conn.rmb_tx_elem->rmb_tx_dma[j];
+			}
+		}
+		if (!smc->conn.rx_rmb_reused) {
+			if (j == i) {
+				rc = smc_map_rmbs_rx(lgr, i,
+						     smc->conn.rmb_rx_size,
+						     smc->conn.rmb_rx_elem);
+				if (rc) {
+					smc_link_down(lgr, i);
+					continue;
+				}
+			} else {
+				smc->conn.rmb_rx_elem->rmb_rx_dma[i] =
+					smc->conn.rmb_rx_elem->rmb_rx_dma[j];
+			}
+			rc = smc_get_dma_mr(lgr, i, smc->conn.rmb_rx_elem);
+			if (rc) {
+				smc_link_down(lgr, i);
+				continue;
+			}
+		}
+	}
+	return rc;
+}
+
+int smc_create_rmbs(struct smc_sock *smc)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_link_group *lgr = conn->lgr;
+	struct rmb_tx_addrs *tx_rmb = NULL;
+	struct rmb_rx_addrs *rx_rmb = NULL;
+	int rmem, wmem, tmp_sl, tmp_ss, sl, ss;
+
+	rmem = smc->sk.sk_rcvbuf;
+	wmem = smc->sk.sk_sndbuf;
+	ss = ror8(smc_compress_bufsize(wmem), 4);
+	sl = smc_uncompress_bufsize(rol8(ss, 4));
+	for (tmp_sl = sl, tmp_ss = ss;
+	     tmp_sl >= SMC_MIN_RMBE_SIZE && tmp_ss >= 0;
+	     tmp_sl = tmp_sl >> 1, tmp_ss--) {
+		tx_rmb = smc_get_tx_rmb_slot(lgr, tmp_ss);
+		if (tx_rmb) {
+			conn->tx_rmb_reused = 1;
+		} else {
+			tx_rmb = (struct rmb_tx_addrs *)
+				kzalloc(sizeof(struct rmb_tx_addrs),
+					GFP_KERNEL);
+			if (!tx_rmb)
+				return -ENOMEM;
+			tx_rmb->rmb_tx = kzalloc(tmp_sl,
+						 GFP_KERNEL | __GFP_NOWARN |
+						 __GFP_NOMEMALLOC |
+						 __GFP_NORETRY);
+			tx_rmb->used = 1;
+			if (tx_rmb->rmb_tx) {
+				list_add(&tx_rmb->list,
+					 &lgr->rmb_tx_bufs[tmp_ss]);
+			}
+		}
+		if (tx_rmb->rmb_tx) {
+			conn->rmb_tx_elem = tx_rmb;
+			memset(conn->rmb_tx_elem->rmb_tx, 0, tmp_sl);
+			break;
+		}
+	}
+	if (!conn->rmb_tx_elem) {
+		kfree(tx_rmb);
+		return -ENOMEM;
+	}
+	conn->rmb_tx_size = tmp_sl;
+	conn->rmb_tx_size_short = rol8(tmp_ss, 4);
+	conn->upd_cons_limit =
+		min_t(int, conn->rmb_rx_size / 10, SOCK_MIN_SNDBUF / 2);
+
+	ss = ror8(smc_compress_bufsize(rmem), 4);
+	sl = smc_uncompress_bufsize(rol8(ss, 4));
+	for (tmp_sl = sl, tmp_ss = ss;
+	     tmp_sl >= SMC_MIN_RMBE_SIZE && tmp_ss >= 0;
+	     tmp_sl = tmp_sl >> 1, tmp_ss--) {
+		rx_rmb = smc_get_rx_rmb_slot(lgr, tmp_ss);
+		if (rx_rmb) {
+			conn->rx_rmb_reused = 1;
+		} else {
+			rx_rmb = (struct rmb_rx_addrs *)
+				kzalloc(sizeof(struct rmb_rx_addrs),
+					GFP_KERNEL);
+			if (!rx_rmb)
+				break;
+			rx_rmb->rmb_rx = kzalloc(tmp_sl,
+						 GFP_KERNEL | __GFP_NOWARN |
+						 __GFP_NOMEMALLOC);
+			rx_rmb->used = 1;
+			if (rx_rmb->rmb_rx)
+				list_add(&rx_rmb->list,
+					 &lgr->rmb_rx_bufs[tmp_ss]);
+		}
+		if (rx_rmb && rx_rmb->rmb_rx) {
+			conn->rmb_rx_elem = rx_rmb;
+			memset(conn->rmb_rx_elem->rmb_rx, 0, tmp_sl);
+			break;
+		}
+	}
+	if (!conn->rmb_rx_elem) {
+		cmpxchg(&tx_rmb->used, 1, 0);
+		conn->rmb_tx_size = 0;
+		conn->rmb_tx_size_short = 0;
+		kfree(rx_rmb);
+		return -ENOMEM;
+	}
+	conn->rmb_rx_size = tmp_sl;
+	conn->rmb_rx_size_short = rol8(tmp_ss, 4);
+
+	return 0;
+}
+
+static struct smc_link *smc_link_match(struct smc_link_group *lgr,
+				       struct smc_clc_msg_hdr2 *hdr2,
+				       u32 qp_num, enum smc_role role,
+				       unsigned long *flags)
+{
+	int i;
+
+	if (mutex_is_locked(&lgr->conf_mutex)) {
+		write_unlock_bh(&lgr->conns_lock);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		mutex_lock(&lgr->conf_mutex);
+		mutex_unlock(&lgr->conf_mutex);
+		spin_lock_bh(&smc_lgr_list.lock);
+		write_lock_bh(&lgr->conns_lock);
+		return (struct smc_link *)LGR_RECONFIGURING;
+	}
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) &&
+		    !memcmp(lgr->lnk[i].gid_peer, &hdr2->gid, 16) &&
+		    !memcmp(lgr->lnk[i].mac_peer, hdr2->mac, 6) &&
+		    ((role == SMC_SERV) ||
+		     (lgr->lnk[i].qp_num_peer == qp_num)))
+			return &lgr->lnk[i];
+	}
+	return NULL;
+}
+
+static int smc_find_link(struct smc_link_group *lgr, int alt)
+{
+	int first = -ENOENT;
+	int second = -ENOENT;
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			first = i;
+			break;
+		}
+	}
+	if (!alt)
+		return first;
+	second = first;
+	for (i = ++first; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			second = i;
+			break;
+		}
+	}
+	return second;
+}
+
+static void smc_link_up_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr
+		= container_of(work, struct smc_link_group, link_up_work);
+
+	if (lgr->lnk_up) {
+		mutex_lock(&lgr->conf_mutex);
+		lgr->lnk_up = 0;
+		smc_link_up(lgr);
+		mutex_unlock(&lgr->conf_mutex);
+	}
+}
+
+static void smc_link_down_worker(struct work_struct *work)
+{
+	int i;
+
+	struct smc_link_group *lgr
+		= container_of(work, struct smc_link_group, link_down_work);
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk_down[i]) {
+			lgr->lnk_down[i] = 0;
+			if (smc_lnk_downing(&lgr->lnk[i].state)) {
+				llc_cancel_wait(&lgr->lnk[i]);
+				mutex_lock(&lgr->conf_mutex);
+				smc_link_down(lgr, i);
+				mutex_unlock(&lgr->conf_mutex);
+			}
+		}
+	}
+}
+
+static int smc_create_lgr(struct smc_sock *smc, __be32 target,
+			  enum smc_role role, struct smc_clc_msg_hdr2 *hdr2)
+{
+	struct smc_link_group *lgr;
+	int i, j, rc;
+	char wq_name[11];
+
+	/* create a new link group */
+	lgr = kzalloc(sizeof(*lgr), GFP_KERNEL);
+	if (!lgr)
+		return -ENOMEM;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		rc = smc_alloc_link_mem(&lgr->lnk[i]);
+		if (rc) {
+			for (j = 0; j < i; j++)
+				smc_free_link_mem(&lgr->lnk[j]);
+			kfree(lgr);
+			return -ENOMEM;
+		}
+		lgr->lnk[i].lnk_grp = lgr;
+	}
+	for (i = 0; i < 16; i++) {
+		INIT_LIST_HEAD(&lgr->rmb_tx_bufs[i]);
+		INIT_LIST_HEAD(&lgr->rmb_rx_bufs[i]);
+	}
+	memcpy(lgr->peer_peer_id, hdr2->peer_id, 8);
+	lgr->daddr = target;
+	lgr->role = role;
+	lgr->lgr_type = INIT;
+	spin_lock_init(&lgr->llc_ctl_lock);
+	INIT_LIST_HEAD(&lgr->llc_event_q);
+	spin_lock_init(&lgr->llc_event_q_lock);
+	lgr->delayed_q = NULL;
+	init_waitqueue_head(&lgr->llc_waiter);
+	INIT_WORK(&lgr->llc_event_work, llc_event_worker);
+	INIT_DELAYED_WORK(&lgr->free_lgr_work, smc_free_lgr_worker);
+	INIT_DELAYED_WORK(&lgr->free_lgr_final_work, smc_free_lgr_final_worker);
+	INIT_WORK(&lgr->llc_add_link_work, llc_process_add_link);
+	INIT_WORK(&lgr->llc_del_link_work, llc_process_del_link);
+	INIT_WORK(&lgr->llc_conf_rkey_work, llc_process_confirm_rkey);
+	INIT_WORK(&lgr->llc_del_rkey_work, llc_process_delete_rkey);
+	lgr->conns_all = RB_ROOT;
+	rwlock_init(&lgr->conns_lock);
+	lgr->lgr_id = atomic_add_return(256, &smc_lgr_num);
+	smc->conn.lgr = lgr;
+	write_lock_bh(&lgr->conns_lock);
+	smc_register_in_lgr(&smc->conn);
+	write_unlock_bh(&lgr->conns_lock);
+	mutex_init(&lgr->conf_mutex);
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_add_tail(&lgr->list, &smc_lgr_list.list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+	mutex_lock(&lgr->conf_mutex);
+	smc->conn.lnk_idx = 0;
+	snprintf(wq_name, sizeof(wq_name), "smcwq%u", lgr->lgr_id);
+	lgr->llc_wq = create_singlethread_workqueue(wq_name);
+	INIT_WORK(&lgr->link_up_work, smc_link_up_worker);
+	INIT_WORK(&lgr->link_down_work, smc_link_down_worker);
+	return 0;
+}
+
+static int smc_check_lgr_consistency(struct smc_sock *smc)
+{
+	u32 reason;
+	u8 flag;
+
+	if (!smc->conn.lgr && !smc->new_lgr) {
+		/* send decline with flag */
+		reason = SMC_CLC_DEC_SU; /* synchronization error */
+		flag = SMC_LINK_GROUP_OUT_OF_SYNCH;
+		clc_send_decline(smc, reason, flag);
+		smc->sk.sk_err = ENOLINK;
+		return -ENOLINK;
+	}
+	if (smc->conn.lgr && smc->new_lgr) {
+		/* something is wrong with this link group */
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_del_init(&smc->conn.lgr->list);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		smc_unregister_in_lgr(&smc->conn); /* takes conns_lock */
+		/* send decline with flag */
+		reason = SMC_CLC_DEC_SU; /* synchronization error */
+		flag = SMC_LINK_GROUP_OUT_OF_SYNCH;
+		clc_send_decline(smc, reason, flag);
+		/* terminate connections */
+		smc->sk.sk_err = ENOLINK;
+		smc_terminate_conn(smc->conn.lgr);
+		return -ENOLINK;
+	}
+	return 0;
+}
+
+int smc_create_conn(struct smc_sock *smc, __be32 target, enum smc_role role,
+		    struct smc_roce_defs *rocdefs,
+		    struct smc_clc_msg_hdr2 *hdr2,
+		    u32 arrival_qp_num)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_link_group *lgr;
+	struct smc_link *link;
+	unsigned long flags;
+	int alt = 0, rc;
+
+	while (atomic_read(&smc_reconfiguring) ||
+	       (cmpxchg(&smc_lgr_list.create_conn_pending, 0, 1) != 0))
+		msleep(20);
+	spin_lock_bh(&smc_lgr_list.lock);
+again:
+	list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+		write_lock_bh(&lgr->conns_lock);
+		if ((!memcmp(lgr->peer_peer_id, hdr2->peer_id, 8)) &&
+		    (lgr->role == role) &&
+		    (lgr->vlan == rocdefs->vlan) &&
+		    ((role == SMC_CLNT) ||
+		     (lgr->conns_num < smc_max_conn_per_lgr))) {
+			link = smc_link_match(lgr, hdr2, arrival_qp_num, role,
+					      &flags);
+			if (!link) {
+				write_unlock_bh(&lgr->conns_lock);
+				continue;
+			}
+			if (link == (struct smc_link *)LGR_RECONFIGURING) {
+				write_unlock_bh(&lgr->conns_lock);
+				goto again;
+			}
+			conn->lgr = lgr;
+			smc_lgr_list.create_conn_pending = 0;
+			if (role == SMC_SERV) {
+				if (lgr->lgr_type == SYMMETRIC)
+					/* try load balancing */
+					alt = lgr->conns_num % 2;
+				conn->lnk_idx = smc_find_link(lgr, alt);
+			} else {
+				conn->lnk_idx = link - &lgr->lnk[0];
+			}
+			smc_register_in_lgr(conn);
+			write_unlock_bh(&lgr->conns_lock);
+			break;
+		}
+		write_unlock_bh(&lgr->conns_lock);
+	}
+	spin_unlock_bh(&smc_lgr_list.lock);
+	if (role == SMC_CLNT) {
+		rc = smc_check_lgr_consistency(smc);
+		if (rc) {
+			smc_lgr_list.create_conn_pending = 0;
+			return -ENOLINK;
+		}
+	}
+	if (!conn->lgr) {
+		smc->new_lgr = 1;
+		rc = smc_create_lgr(smc, target, role, hdr2);
+		smc_lgr_list.create_conn_pending = 0;
+		if (rc)
+			return rc;
+		lgr = conn->lgr;
+		rc = smc_create_link(lgr, 0, rocdefs);
+		if (rc) {
+			lgr->lgr_type = NONE;
+			mutex_unlock(&lgr->conf_mutex);
+			spin_lock_bh(&smc_lgr_list.lock);
+			list_del(&lgr->list);
+			spin_unlock_bh(&smc_lgr_list.lock);
+			kfree(lgr);
+			return rc;
+		}
+		link = &lgr->lnk[conn->lnk_idx];
+	} else {
+		mutex_lock(&lgr->conf_mutex);
+		smc->new_lgr = 0;
+	}
+	conn->local_tx_ctrl.ctrl_type = RMBE_CTRL;
+	conn->local_tx_ctrl.ctrl_len = SMC_SIZE_OF_CTRL_DATA;
+	conn->tx_seq = 0;
+	spin_lock_init(&conn->send_lock);
+	atomic_set(&conn->bytes_to_rcv, 0);
+	conn->lgr->vlan = rocdefs->vlan;
+	init_waitqueue_head(&smc->rx_waiter);
+	return 0;
+}
+
+u8 smc_compress_bufsize(int n)
+{
+	u8 compressed = 0;
+
+	n = (n - 1) >> 14;
+	compressed = ilog2(n) + 1;
+	if (compressed > SMC_RMBE_SIZE_VALUES)
+		compressed = SMC_RMBE_SIZE_VALUES;
+	compressed = rol8(compressed, 4);
+	return compressed;
+}
+
+int smc_uncompress_bufsize(u8 x)
+{
+	u8 left_bits = ror8(x, 4);
+	u32 size;
+
+	size = 0x00000001 << (((int)left_bits) + 14);
+	return size;
+}
+
+static int smc_rdma_write(struct smc_connection *conn, int target_offset,
+			  int lnk_idx, int num, struct tx_sge sge[])
+{
+	struct smc_link_group *lgr = conn->lgr;
+	struct smc_link *link;
+	struct ib_sge sge_send[4];
+	struct ib_send_wr send_wr;
+	struct ib_send_wr *failed_wr;
+	int i, j, rc;
+	int to = target_offset;
+
+	memset(&send_wr, 0, sizeof(send_wr));
+	memset(&sge_send, 0, 4 * sizeof(struct ib_sge));
+	link = &lgr->lnk[lnk_idx];
+	for (i = 0, j = 0; (i < num) && (j < 4); i++, j++) {
+		if ((to % lgr->cache_align_size) &&
+		    ((to / lgr->cache_align_size)) !=
+		     ((to + sge[i].len - 1) / lgr->cache_align_size)) {
+			int initial = lgr->cache_align_size -
+				      to % lgr->cache_align_size;
+
+			/* split into aligned and unaligned part */
+			sge_send[j].addr =
+				((u64)conn->rmb_tx_elem->rmb_tx_dma[lnk_idx]) +
+				sge[i].offset;
+			sge_send[j].length = initial;
+			sge_send[j].lkey = link->mr_tx->lkey;
+			to += initial;
+			sge[i].len -= initial;
+			sge[i].offset += initial;
+			j++;
+		}
+		sge_send[j].addr = ((u64)conn->rmb_tx_elem->rmb_tx_dma[lnk_idx])
+				 + sge[i].offset;
+		sge_send[j].length = sge[i].len;
+		sge_send[j].lkey = link->mr_tx->lkey;
+	}
+	send_wr.wr_id =
+		(u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	send_wr.opcode = IB_WR_RDMA_WRITE;
+	send_wr.wr.rdma.remote_addr =
+		lgr->rtok[conn->rtok_idx].vaddr[lnk_idx]
+	      + target_offset
+	      + ((conn->peer_tcp_conn_idx - 1) * (conn->peer_rx_buf_len));
+	send_wr.wr.rdma.rkey = lgr->rtok[conn->rtok_idx].rkey[lnk_idx];
+	send_wr.num_sge = j;
+	send_wr.sg_list = sge_send;
+	failed_wr = &send_wr;
+	rc = ib_post_send(link->roce_qp, &send_wr, &failed_wr);
+	if (rc) {
+		struct smc_roce_defs *rocd;
+
+		rocd = &link->roce_defs;
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+			if (cmpxchg(&rocd->ibdev->port_err[rocd->port - 1],
+				    0, 1) == 0)
+				schedule_work(&rocd->ibdev->port_err_work);
+	}
+	return rc;
+}
+
+static inline int smc_get_free_ctrl_buf(struct smc_link *link, u32 *elem)
+{
+	*elem = link->send_wr_num;
+	if ((atomic_read(&link->state) == SMC_LINK_DOWN) ||
+	    (atomic_read(&link->state) == SMC_LINK_FREED)) {
+		return -EPIPE;
+	}
+	for_each_clear_bit(*elem, link->pending_bit_mask, link->send_wr_num) {
+		if (!test_and_set_bit(*elem, link->pending_bit_mask))
+			return 0;
+	}
+	*elem = link->send_wr_num;
+	return -EBUSY;
+}
+
+int smc_get_ctrl_buf(struct smc_link *link, u32 *elem)
+{
+	int rc = 0;
+
+	if (in_softirq()) {
+		rc = smc_get_free_ctrl_buf(link, elem);
+		if (rc)
+			return rc;
+	} else {
+		rc = wait_event_interruptible_timeout(link->wqe_waiter,
+				(smc_get_free_ctrl_buf(link, elem) != -EBUSY),
+				SMC_WAIT_FREE_CTRL_BUF_TIMEO);
+		if (!rc)
+			return -EPIPE;
+		if (rc == -ERESTARTSYS)
+			return -EINTR;
+		if (*elem == link->send_wr_num)
+			return -EPIPE;
+	}
+	return 0;
+}
+
+static inline int smc_add_pending_send(struct smc_link *link,
+				       struct smc_connection *conn, u32 *elem)
+{
+	u64 wr_id;
+	int rc = 0;
+
+	wr_id = (u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	rc = smc_get_ctrl_buf(link, elem);
+	if (rc < 0)
+		return rc;
+	memset(&link->pending[*elem], 0, sizeof(struct pending_send));
+	link->pending[*elem].conn = conn;
+	link->pending[*elem].wr_id_send = wr_id;
+	link->send_wr[*elem].wr_id = wr_id;
+	return 0;
+}
+
+int smc_ctrl_send(struct smc_connection *conn, int lnk_idx)
+{
+	struct smc_link *link;
+	struct rmb_e_ctrl ctrl;
+	int rc = 0;
+	u32 elem;
+
+	link = &conn->lgr->lnk[lnk_idx];
+	rc = smc_add_pending_send(link, conn, &elem);
+	if (rc)
+		return rc;
+	spin_lock_bh(&conn->send_lock);
+	if (conn->local_tx_ctrl.p_flags.last_msg_valid_req) {
+		conn->local_tx_ctrl.ctrl_seq = conn->tx_seq_fin;
+	} else {
+		conn->tx_seq++;
+		conn->local_tx_ctrl.ctrl_seq = conn->tx_seq;
+	}
+	link->pending[elem].cursor.s.curs = conn->tx_curs_sent.s.curs;
+	link->pending[elem].p_cursor.s.curs =
+		conn->local_tx_ctrl.p_curs.s.curs;
+	link->pending[elem].ctrl_seq = conn->tx_seq;
+	smc_set_ctrl(&conn->local_tx_ctrl, &ctrl);
+	rc = smc_wr_send(link, conn, (u64)&ctrl, elem);
+	if (!rc) {
+		atomic64_set(&conn->rx_curs_confirmed.s.acurs,
+			    atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs));
+	} else {
+		test_and_clear_bit(elem, link->pending_bit_mask);
+	}
+	spin_unlock_bh(&conn->send_lock);
+	return rc;
+}
+
+/* Send wr via ib_post_send.
+ * Requires conn->send_lock being held if entered with an smc_connection
+ */
+int smc_wr_send(struct smc_link *link, struct smc_connection *conn, u64 addr,
+		u32 index)
+{
+	struct ib_send_wr *failed_wr;
+	struct smc_roce_defs *rocdefs = &link->roce_defs;
+	struct smc_ib_device *ibdev = rocdefs->ibdev;
+	int rc;
+
+	rc = ib_req_notify_cq(ibdev->roce_cq_send,
+			      IB_CQ_SOLICITED_MASK |
+			      IB_CQ_REPORT_MISSED_EVENTS);
+	memcpy(&link->ctrl_buf_tx[index], (void *)addr, SMC_SIZE_OF_CTRL_DATA);
+	failed_wr = &link->send_wr[index];
+
+	rc = ib_post_send(link->roce_qp, &link->send_wr[index], &failed_wr);
+	if (rc) {
+		if ((atomic_read(&link->state) != SMC_LINK_FREED) &&
+		    (cmpxchg(&ibdev->port_err[rocdefs->port - 1], 0, 1) == 0))
+				/* try failover to other link */
+			schedule_work(&ibdev->port_err_work);
+		if (conn) {
+			if (!conn->local_rx_ctrl.p_flags.write_blocked &&
+			    !conn->local_rx_ctrl.p_flags.cons_curs_upd_req) {
+				/* not invoked in softirq context */
+				spin_unlock_bh(&conn->send_lock);
+				msleep(100); /* delay to enable port_err_work
+					      * to be done
+					      */
+				spin_lock_bh(&conn->send_lock);
+			}
+		} else {
+			msleep(100);
+		}
+		rc = 0;
+	}
+	return rc;
+}
+
+static int smc_write_space(struct smc_sock *smc)
+{
+	struct smc_curs prod;
+	struct smc_curs cons;
+	int buffer_len = smc->conn.peer_rx_buf_len;
+	int space;
+
+	prod.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&smc->conn.local_rx_ctrl.c_curs.s.acurs);
+
+	/* determine rx_buf space */
+	space = buffer_len - smc_curs_diff(buffer_len, &cons, &prod);
+
+	return space;
+}
+
+static int smc_wait_rx_space(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int space, rc = 0;
+
+	space = atomic_read(&conn->rx_buf_space);
+	if (space)
+		goto out;
+
+	release_sock(sk);
+	rc = wait_event_interruptible_timeout(smc->rx_waiter,
+		atomic_read(&conn->rx_buf_space) ||
+		sk->sk_err ||
+		smc_stop_received(conn),
+		SMC_WAIT_RX_SPACE_TIMEO);
+	if ((rc == -ERESTARTSYS) || !rc) {
+		/* interrupted or timeout */
+		smc_flag_sock_err(smc);
+		space = sock_error(sk);
+		lock_sock(sk);
+		goto out;
+	}
+	lock_sock(sk);
+	space = atomic_read(&conn->rx_buf_space);
+out:
+	return space;
+}
+
+static int smc_wait_tx_space(struct smc_sock *smc, int flags)
+{   /* derived from sk_stream_wait_memory */
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT);
+	int rc = 0;
+	DEFINE_WAIT(wait);
+
+	if (timeo == MAX_SCHEDULE_TIMEOUT)
+		timeo = (prandom_u32() % (HZ / 5)) + 2;
+	set_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+	if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN) ||
+	    conn->local_tx_ctrl.conn_state_flags.sending_done) {
+		rc = -EPIPE;
+		goto out;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ECONNRESET;
+		goto out;
+	}
+	if (!timeo) {
+		rc = -EAGAIN;
+		goto out;
+	}
+	if (signal_pending(current)) {
+		rc = -EINTR;
+		goto out;
+	}
+	clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
+	if (atomic_read(&conn->tx_buf_space))
+		goto out;
+	set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+	rc = sk_wait_event(sk, &timeo, sk->sk_err ||
+			   (sk->sk_shutdown & SEND_SHUTDOWN) ||
+			   conn->local_tx_ctrl.conn_state_flags.sending_done ||
+			   smc_close_received(conn) ||
+			   atomic_read(&conn->tx_buf_space));
+	clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+	if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN) ||
+	    conn->local_tx_ctrl.conn_state_flags.sending_done) {
+		rc = -EPIPE;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		rc = -ECONNRESET;
+out:
+	finish_wait(sk_sleep(sk), &wait);
+	return rc;
+}
+
+int smc_prepared_sends(struct smc_connection *conn)
+{
+	struct smc_curs prep;
+	struct smc_curs sent;
+
+	prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	sent.s.lcurs = atomic64_read(&conn->tx_curs_sent.s.acurs);
+	return smc_curs_diff(conn->rmb_tx_size, &sent, &prep);
+}
+
+int smc_pending_sends(struct smc_connection *conn)
+{
+	struct smc_link *lnk;
+	u32 i, j;
+
+	if ((!conn) || (!conn->lgr))
+		return 0;
+
+	for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+		lnk = &conn->lgr->lnk[j];
+		for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+			if (lnk->pending[i].conn == conn)
+				return 1;
+		}
+	}
+	return 0;
+}
+
+void smc_clear_pending_sends(struct smc_connection *conn)
+{
+	struct smc_link *lnk;
+	u32 i, j;
+
+	for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+		lnk = &conn->lgr->lnk[j];
+		for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+			if (lnk->pending[i].conn == conn)
+				clear_bit(i, lnk->pending_bit_mask);
+		}
+	}
+}
+
+static inline int smc_give_up_send(struct smc_sock *smc, int copied)
+{
+	struct smc_connection *conn = &smc->conn;
+
+	if (smc->sk.sk_shutdown & SEND_SHUTDOWN ||
+	    conn->local_tx_ctrl.conn_state_flags.abnormal_close)
+		return -EPIPE;
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		return copied ? copied : -ECONNRESET;
+	if (conn->local_rx_ctrl.conn_state_flags.closed_conn)
+		return copied ? copied : -ECONNRESET;
+	return 0;
+}
+
+int smc_conn_send(struct smc_sock *smc, struct msghdr *msg, size_t len)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct smc_curs prep;
+	unsigned long max_wait_for_tx_space;
+	int copy, copied = 0, to_send = len;
+	int writespace;
+	int tx_top = 0, tx_bot = 0;
+	int rc = 0;
+	int tx_cnt_prep;
+	char *buf;
+
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+	}
+	max_wait_for_tx_space =  SMC_WAIT_TX_SPACE_TIMEO + jiffies;
+again:
+	if ((smc->sk.sk_state != SMC_ACTIVE) &&
+	    (smc->sk.sk_state != SMC_APPLCLW1))
+		return (smc->sk.sk_state == SMC_INIT
+		      ? -ENOTCONN : -ECONNRESET);
+
+	writespace = atomic_read(&conn->tx_buf_space);
+	if (!writespace) {
+		writespace = smc_wait_tx_space(smc, msg->msg_flags);
+		if (writespace < 0)
+			return ((copied && (writespace == -EAGAIN))
+			       ? copied : writespace);
+		if (!writespace) {
+			rc = smc_give_up_send(smc, copied);
+			if (rc)
+				return rc;
+			if (msg->msg_flags & MSG_DONTWAIT)
+				return copied ? copied : -EAGAIN;
+			if ((smc->sk.sk_err) ||
+			    (time_after_eq(jiffies, max_wait_for_tx_space)))
+				return copied ? copied : -EPIPE;
+			if (smc_check_peer_gone(smc->conn.lgr)) {
+				smc->sk.sk_err = EPIPE;
+				release_sock(&smc->sk);
+				smc_terminate_conn(smc->conn.lgr);
+				lock_sock(&smc->sk);
+				return copied ? copied : -EPIPE;
+			}
+			goto again;
+		}
+	}
+	if (smc->sk.sk_err ||
+	    smc->sk.sk_shutdown & SEND_SHUTDOWN)
+		return -EPIPE;
+
+	writespace = atomic_read(&conn->tx_buf_space);
+	rc = smc_give_up_send(smc, copied);
+	if (rc)
+		return rc;
+
+	if (msg->msg_flags & MSG_OOB) {
+		if (writespace < len) {
+			conn->local_tx_ctrl.p_flags.urg_data_pending = 1;
+			rc = smc_ctrl_send(conn, conn->lnk_idx);
+			conn->local_tx_ctrl.p_flags.urg_data_pending = 0;
+		}
+	}
+
+	copy = min_t(size_t, to_send, writespace);
+	/* determine start of tx_buf */
+	prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	tx_cnt_prep = prep.s.curs.c;
+	buf = conn->rmb_tx_elem->rmb_tx;
+	/* determine tx_buf pieces - bottom and top of tx_buf */
+	if (tx_cnt_prep + copy <= conn->rmb_tx_size) {
+		tx_top = 0;
+		tx_bot = copy;
+		if (memcpy_from_msg(buf + tx_cnt_prep, msg, copy))
+			return -EFAULT;
+	} else {
+		tx_bot = conn->rmb_tx_size - tx_cnt_prep;
+		tx_top = copy - tx_bot;
+		if (memcpy_from_msg(buf + tx_cnt_prep, msg, tx_bot))
+			return -EFAULT;
+		if (memcpy_from_msg(buf, msg, tx_top))
+			return -EFAULT;
+	}
+	smc_curs_add(conn->rmb_tx_size, &prep.s.curs, copy);
+	atomic64_set(&conn->tx_curs_prep.s.acurs, prep.s.lcurs);
+	smp_mb__before_atomic();
+	atomic_sub(copy, &conn->tx_buf_space);
+	smp_mb__after_atomic();
+
+	if (msg->msg_flags & MSG_OOB)
+		atomic64_set(&conn->tx_urg_curs.s.acurs, prep.s.lcurs);
+
+	if ((tp->nonagle & TCP_NAGLE_CORK || msg->msg_flags & MSG_MORE) &&
+	    (writespace > copy)) {
+		if (!timer_pending(&smc->cork_timer))
+			/* cancelled at the beginning ? */
+			mod_timer(&smc->cork_timer,
+				  jiffies + msecs_to_jiffies(200));
+	} else {
+		if (atomic_read(&conn->rx_buf_space))
+			smc_write_data(smc);
+		else if (!delayed_work_pending(&smc->write_work))
+			schedule_delayed_work(&smc->write_work, 0);
+	}
+
+	copied = copied + copy;
+	to_send = to_send - copy;
+	if (copied < len)
+		goto again;
+
+	return copied;
+}
+
+static inline void smc_fill_sge(int *num, struct tx_sge sge[], int offset1,
+				int len1, int offset2, int len2)
+{
+	memset(sge, 0, 2 * sizeof(struct tx_sge));
+	sge[0].offset = offset1;
+	sge[0].len = len1;
+	if (len2) {
+		*num = 2;
+		sge[1].offset = offset2;
+		sge[1].len = len2;
+	} else {
+		*num = 1;
+	}
+}
+
+static inline void smc_write_data_adapt(struct smc_connection *conn,
+					struct smc_curs *prod,
+					struct smc_curs *sent, int len)
+{
+	smc_curs_add(conn->peer_rx_buf_len, &prod->s.curs, len);
+	smp_mb__before_atomic();
+	atomic_sub(len, &conn->rx_buf_space);
+	smp_mb__after_atomic();
+	smc_curs_add(conn->rmb_tx_size, &sent->s.curs, len);
+}
+
+void smc_write_data(struct smc_sock *smc)
+{
+	struct smc_link *lnk;
+	struct smc_connection *conn = &smc->conn;
+	int copy, space_left1, space_left2, send_len, to_send;
+	int writespace;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs sent;
+	struct smc_curs prep;
+	struct tx_sge sge[2];
+	int tx_top = 0, tx_bot = 0;
+	int tx_top1 = 0, tx_top2 = 0;
+	int tx_bot1 = 0, tx_bot2 = 0;
+	int rc = 0, num, urgent = 0;
+	u8 lnk_idx;
+
+again:
+	lnk_idx = conn->lnk_idx;
+	if (conn->lgr->lnk_down[lnk_idx]) {
+		release_sock(&smc->sk);
+		msleep(1000);
+		lock_sock(&smc->sk);
+		goto again;
+	}
+	lnk = &conn->lgr->lnk[lnk_idx];
+	sent.s.lcurs = atomic64_read(&conn->tx_curs_sent.s.acurs);
+	prep.s.lcurs = atomic64_read(&conn->tx_urg_curs.s.acurs);
+	if (!prep.s.curs.w && !prep.s.curs.c)
+		prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	else
+		urgent = 1;
+	to_send = smc_curs_diff(conn->rmb_tx_size, &sent, &prep);
+	if (!to_send)
+		goto send_ctrl;
+	writespace = atomic_read(&conn->rx_buf_space);
+	if (writespace <= 0)
+		return;
+	if (find_first_zero_bit(lnk->pending_bit_mask, lnk->send_wr_num) ==
+		lnk->send_wr_num) {
+		schedule_delayed_work(&smc->write_work, (HZ / 10));
+		return;
+	}
+	copy = min_t(size_t, to_send, writespace);
+	if (sent.s.curs.c + copy <= conn->rmb_tx_size) {
+		tx_top = 0;
+		tx_bot = copy;
+	} else {
+		tx_bot = conn->rmb_tx_size - sent.s.curs.c;
+		tx_top = copy - tx_bot;
+	}
+	prod.s.lcurs = atomic64_read(&conn->local_tx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left1 = conn->peer_rx_buf_len - prod.s.curs.c;
+		space_left2 = cons.s.curs.c;
+
+		send_len = min_t(size_t, copy, space_left1);
+		if (send_len <= tx_bot) {
+			tx_bot1 = send_len;
+			tx_bot2 = tx_bot - tx_bot1;
+			tx_top1 = 0;
+			tx_top2 = tx_top;
+		} else {
+			tx_bot1 = tx_bot;
+			tx_bot2 = 0;
+			tx_top1 = send_len - tx_bot;
+			tx_top2 = tx_top - tx_top1;
+		}
+		smc_fill_sge(&num, sge, sent.s.curs.c, tx_bot1, 0, tx_top1);
+		rc = smc_rdma_write(conn, prod.s.curs.c, lnk_idx, num, sge);
+		if (rc)
+			return;
+		copy -= send_len;
+		smc_write_data_adapt(conn, &prod, &sent, send_len);
+
+		if (copy && space_left2 && (tx_bot2 + tx_top2 > 0)) {
+			send_len = min_t(size_t, copy, space_left2);
+			if (tx_bot2 > send_len) {
+				tx_bot2 = send_len;
+				tx_top2 = 0;
+			} else {
+				if (tx_bot2 + tx_top2 > send_len)
+					tx_top2 = send_len - tx_bot2;
+			}
+			if (tx_bot2)
+				smc_fill_sge(&num, sge, sent.s.curs.c,
+					     tx_bot2, tx_top1, tx_top2);
+			else if (tx_top2)
+				smc_fill_sge(&num, sge, tx_top1, tx_top2, 0, 0);
+			rc = smc_rdma_write(conn, 0, lnk_idx, num, sge);
+			if (rc)
+				return;
+			smc_write_data_adapt(conn, &prod, &sent,
+					     tx_bot2 + tx_top2);
+		}
+	} else {
+		space_left1 = cons.s.curs.c - prod.s.curs.c;
+		send_len = min_t(size_t, copy, space_left1);
+		if (send_len <= tx_bot) {
+			tx_bot = send_len;
+			tx_top = 0;
+		} else {
+			if ((send_len - tx_bot) <= tx_top)
+				tx_top = send_len - tx_bot;
+		}
+		smc_fill_sge(&num, sge, sent.s.curs.c, tx_bot, 0, tx_top);
+		rc = smc_rdma_write(conn, prod.s.curs.c, lnk_idx, num, sge);
+		if (rc)
+			return;
+		smc_write_data_adapt(conn, &prod, &sent, send_len);
+	}
+	atomic64_set(&conn->local_tx_ctrl.p_curs.s.acurs, prod.s.lcurs);
+	atomic64_set(&conn->tx_curs_sent.s.acurs, sent.s.lcurs);
+
+	/* send cursor updates and signal peer */
+	if (to_send > atomic_read(&conn->rx_buf_space))
+		conn->local_tx_ctrl.p_flags.write_blocked = 1;
+	else
+		conn->local_tx_ctrl.p_flags.write_blocked = 0;
+
+	if (urgent) {
+		conn->local_tx_ctrl.p_flags.urg_data_pending = 1;
+		conn->local_tx_ctrl.p_flags.urg_data_present = 1;
+	}
+
+send_ctrl:
+	rc = smc_ctrl_send(conn, lnk_idx);
+	if (rc)
+		/* failover is scheduled */
+		return;
+
+	if (urgent) {
+		conn->local_tx_ctrl.p_flags.urg_data_pending = 0;
+		conn->local_tx_ctrl.p_flags.urg_data_present = 0;
+		atomic64_set(&conn->tx_urg_curs.s.acurs, 0);
+			/* could be lost in case of failover */
+	}
+
+	prep.s.lcurs = atomic64_read(&conn->tx_urg_curs.s.acurs);
+	if (!prep.s.curs.w && !prep.s.curs.c)
+		prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	if (smc_curs_diff(conn->rmb_tx_size, &sent, &prep) &&
+	    (smc->sk.sk_state != SMC_CLOSED)) {
+		schedule_delayed_work(&smc->write_work, 0);
+	} else {
+		if (wq_has_sleeper(smc->sk.sk_wq) &&
+		    atomic_read(&conn->tx_buf_space) &&
+		    test_bit(SOCK_NOSPACE, &smc->sk.sk_socket->flags))
+			smc_sock_wake_tx(&smc->sk);
+		if (smc->sk.sk_state != SMC_ACTIVE)
+			wake_up(&smc->prep_waiter);
+	}
+}
+
+void smc_write_worker(struct work_struct *work)
+{
+	int rc;
+
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+		write_work.work);
+
+	lock_sock(&smc->sk);
+again:
+	if (smc_prepared_sends(&smc->conn) == 0)
+		goto out;
+	rc = smc_wait_rx_space(smc);
+	if (rc < 0)
+		goto out;
+	if (!rc) {
+		if (!smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close &&
+		    !smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close &&
+		    (smc->sk.sk_state != SMC_CLOSED))
+			goto again;
+		else
+			goto out;
+	}
+	if (!smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close &&
+	    !smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close &&
+	    (smc->sk.sk_state != SMC_CLOSED))
+		smc_write_data(smc);
+out:
+	release_sock(&smc->sk);
+}
+
+void smc_send_close_alt(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) &&
+		    (i != smc->conn.lnk_idx))
+			break;
+	}
+	if (i <= SMC_MAX_SYM_LINKS)
+		smc_ctrl_send(&smc->conn, i);
+}
+
+int smc_send_close(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int rc;
+
+	if (smc_to_read(smc) > 0)
+		smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	else
+		smc->conn.local_tx_ctrl.conn_state_flags.closed_conn = 1;
+	if (lgr->lgr_type != NONE) {
+		rc = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		if (rc && (lgr->lgr_type != SINGLE)) {
+			/* try other link */
+			smc_send_close_alt(smc);
+		}
+	}
+	wake_up_interruptible(&smc->rx_waiter);
+	return 0;
+}
+
+int smc_to_read(struct smc_sock *smc)
+{
+	int rc;
+	struct smc_curs prod; /* to receive */
+	struct smc_curs cons; /* already consumed */
+	struct smc_curs spl_cons; /* spliced data in flight */
+	struct smc_curs urg;
+
+	if ((smc->conn.rx_urg_state != SMC_URG_VALID) &&
+	    !smc_splice_in_use(smc))
+		return atomic_read(&smc->conn.bytes_to_rcv);
+
+	/* local rx cursor consumer_c is up to date */
+	prod.s.lcurs = atomic64_read(&smc->conn.local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+
+	if ((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&smc->conn.rx_urg_curs.s.acurs);
+		if (smc_curs_diff(smc->conn.rmb_rx_size, &cons, &urg))
+			prod.s.lcurs =
+				atomic64_read(&smc->conn.rx_urg_curs.s.acurs);
+	}
+	if (smc_splice_in_use(smc)) {
+		spl_cons.s.lcurs =
+			atomic64_read(&smc->conn.local_rx_spliced.s.acurs);
+		if (spl_cons.s.curs.reserved0)
+			cons = spl_cons;
+	}
+
+	rc = smc_curs_diff(smc->conn.rmb_rx_size, &cons, &prod);
+	return (rc == -EALREADY) ? 0 : rc;
+}
+
+static void smc_update_cons_curs(struct smc_sock *smc, struct smc_link *lnk)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0, to_confirm;
+	struct smc_curs cfed;
+	struct smc_curs cons;
+
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	cfed.s.lcurs = atomic64_read(&conn->rx_curs_confirmed.s.acurs);
+	to_confirm = smc_curs_diff(conn->rmb_rx_size, &cfed, &cons);
+
+	if (conn->local_rx_ctrl.p_flags.cons_curs_upd_req ||
+	    ((to_confirm > conn->upd_cons_limit) &&
+	     ((to_confirm / (conn->rmb_rx_size / 2) > 0) ||
+	      conn->local_rx_ctrl.p_flags.write_blocked ||
+	      conn->local_rx_ctrl.p_flags.urg_data_pending))) {
+		if (lnk)
+			rc = smc_ctrl_send(conn, lnk - &conn->lgr->lnk[0]);
+		else
+			rc = smc_ctrl_send(conn, conn->lnk_idx);
+		if (rc)
+			return;
+		atomic64_set(&conn->rx_curs_confirmed.s.acurs,
+			    atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs));
+		conn->local_rx_ctrl.p_flags.cons_curs_upd_req = 0;
+	}
+	if (conn->local_rx_ctrl.p_flags.write_blocked &&
+	    (!smc_to_read(smc) > 0))
+		conn->local_rx_ctrl.p_flags.write_blocked = 0;
+}
+
+static void smc_pipe_buf_noop(struct pipe_inode_info *pipe,
+			      struct pipe_buffer *buf)
+{
+	;
+}
+
+static int smc_pipe_buf_steal(struct pipe_inode_info *pipe,
+			      struct pipe_buffer *buf)
+{
+	return 1;
+}
+
+struct smc_pipe_buf_priv {
+	struct smc_sock *smc;
+	unsigned int     len;
+};
+
+static void smc_pipe_buf_release(struct pipe_inode_info *pipe,
+				 struct pipe_buffer *buf)
+{
+	struct smc_curs curs;
+	struct smc_pipe_buf_priv *priv
+		= (struct smc_pipe_buf_priv *)buf->private;
+	struct smc_sock *smc = priv->smc;
+
+	/* Update cursors when we reached the last page */
+	curs.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+	smc_curs_add(smc->conn.rmb_rx_size, &curs.s.curs, priv->len);
+	atomic64_set(&smc->conn.local_tx_ctrl.c_curs.s.acurs, curs.s.lcurs);
+	smc_update_cons_curs(smc, NULL);
+	wake_up(&smc->splice_waiter);
+	kfree((void *)buf->private);
+}
+
+static const struct pipe_buf_operations smc_pipe_ops = {
+	.can_merge = 0,
+	.confirm = generic_pipe_buf_confirm,
+	.get = smc_pipe_buf_noop,
+	.release = smc_pipe_buf_release,
+	.steal = smc_pipe_buf_steal
+};
+
+static void smc_splice_spd_release(struct splice_pipe_desc *spd, unsigned int i)
+{
+	;
+}
+
+static void smc_free_spd(struct smc_sock *smc)
+{
+	if (smc->spd) {
+		kfree(smc->spd->partial);
+		smc->spd->partial = NULL;
+		kfree(smc->spd->pages);
+		smc->spd->pages = NULL;
+		kfree(smc->spd);
+		smc->spd = NULL;
+	}
+}
+
+static int smc_alloc_spd(struct smc_sock *smc)
+{
+	int msize = smc->conn.rmb_rx_size / PAGE_SIZE + 1;
+
+	smc->spd = kzalloc(sizeof(*smc->spd), GFP_KERNEL);
+	if (!smc->spd)
+		return -ENOMEM;
+	smc->spd->pages = kcalloc(msize, sizeof(struct page), GFP_KERNEL);
+	if (!smc->spd->pages)
+		goto error;
+	smc->spd->partial = kcalloc(msize, sizeof(struct partial_page),
+				    GFP_KERNEL);
+	if (!smc->spd->partial)
+		goto error;
+	smc->spd->nr_pages_max = msize;
+	smc->spd->ops = &smc_pipe_ops;
+	smc->spd->spd_release = smc_splice_spd_release;
+
+	return 0;
+
+error:
+	smc_free_spd(smc);
+	return -ENOMEM;
+}
+
+/* no extra handling of splice options required:
+ *   SPLICE_F_MOVE doesn't apply and can be ignored
+ *   SPLICE_F_NONBLOCK is handled by splice_to_pipe
+ *   SPLICE_F_MORE doesn't apply (fdout is a pipe)
+ *   SPLICE_F_GIFT is unused
+ */
+static int smc_splice_to_pipe(struct smc_sock *smc,
+			      struct pipe_inode_info *pipe,
+			      unsigned int flags, void *src, int len)
+{
+	struct partial_page *ppage;
+	struct splice_pipe_desc *spd;
+	struct smc_pipe_buf_priv *priv;
+	int rc, i;
+
+	if (!len)
+		return 0;
+	if (!smc->spd && smc_alloc_spd(smc))
+		return -ENOMEM;
+	spd = smc->spd;
+	spd->nr_pages = 0;
+	spd->flags = flags;
+
+	while (len) {
+		spd->pages[spd->nr_pages] = virt_to_page(src);
+		ppage = &spd->partial[spd->nr_pages];
+		ppage->offset = (unsigned long)src & ~PAGE_MASK;
+		ppage->len = min_t(int, PAGE_SIZE - ppage->offset, len);
+		src += ppage->len;
+		len -= ppage->len;
+		priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+		if (!priv)
+			goto error;
+		priv->smc = smc;
+		priv->len = ppage->len;
+		ppage->private = (unsigned long)priv;
+		spd->nr_pages++;
+	}
+
+	rc = splice_to_pipe(pipe, spd);
+	if (rc < 0 && smc->sk.sk_err)
+		goto error;
+	return rc;
+
+error:
+	for (i = 0; i < spd->nr_pages; ++i)
+		kfree((void *)spd->partial[i].private);
+	return -EFAULT;
+}
+
+/* Wait for data to arrive
+ *   @smc    smc socket
+ *   @len    num bytes to wait for
+ *   @wall   don't return until full request is fulfilled
+ *   @timeo  max seconds to wait, 0 for no timeout
+ * Returns boolean value or -EAGAIN in case timeout expired.
+ */
+int smc_wait_rx_data(struct smc_sock *smc, int len, bool wall,
+		     long timeo)
+{
+	int rc;
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int target;
+	DEFINE_WAIT(wait);
+
+	if ((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID)
+		target = 1;
+	else
+		target = sock_rcvlowat(sk, wall, len);
+	if (smc_to_read(smc) >= target)
+		return 1;
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+	set_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
+	rc = sk_wait_event(sk, &timeo,
+			   (sk->sk_err != 0) ||
+			   sk->sk_shutdown & RCV_SHUTDOWN ||
+			   sock_flag(sk, SOCK_DONE) ||
+			   (smc_to_read(smc) >= target) ||
+			   smc_stop_received(conn));
+	clear_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
+	finish_wait(sk_sleep(sk), &wait);
+	return ((rc || timeo) ? rc : -EAGAIN);
+}
+
+static inline void smc_skip_urg_data(struct smc_sock *smc,
+				     struct smc_curs *cons,
+				     struct smc_curs *urg)
+{
+	if (((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) &&
+	    (cons->s.curs.c == urg->s.curs.c) &&
+	    (cons->s.curs.w == urg->s.curs.w))
+		smc->conn.rx_urg_state |= SMC_URG_RECV;
+}
+
+/* derived from tcp_recv_urg */
+static int smc_conn_recv_urg(struct smc_sock *smc, struct msghdr *msg,
+			     size_t len, unsigned int flags)
+{
+	int rc = 0;
+
+	if (sock_flag(&smc->sk, SOCK_URGINLINE) ||
+	    !(smc->conn.rx_urg_state & SMC_URG_VALID) ||
+	    (smc->conn.rx_urg_state & SMC_URG_READ))
+		return -EINVAL;
+
+	if (!(flags & MSG_PEEK)) {
+		smc->conn.rx_urg_state |= SMC_URG_READ;
+		atomic_dec(&smc->conn.bytes_to_rcv);
+	}
+
+	msg->msg_flags |= MSG_OOB;
+
+	if (len > 0) {
+		if (!(flags & MSG_TRUNC))
+			rc = memcpy_to_msg(msg, &smc->conn.rx_urg_data, 1);
+		len = 1;
+	} else {
+		msg->msg_flags |= MSG_TRUNC;
+	}
+	if (!sock_flag(&smc->sk, SOCK_URGINLINE)) {
+		struct smc_curs cons;
+
+		/* may cause smc_curs_diff errors, which are acceptable here */
+		cons.s.lcurs =
+			atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+		smc_curs_add(smc->conn.rmb_rx_size, &cons.s.curs, 1);
+		atomic64_set(&smc->conn.local_tx_ctrl.c_curs.s.acurs,
+			     cons.s.lcurs);
+	}
+
+	return rc ? -EFAULT : len;
+}
+
+int smc_conn_recv(struct smc_sock *smc, struct msghdr *msg, size_t len,
+		  int flags)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+	int toread = 0, space_left, copy, buffer_len;
+	int target;		/* Read at least these many bytes */
+	char *buffer;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs urg;
+	struct smc_curs spl_cons;
+	long timeo;
+	int loop = 0;
+
+	msg->msg_namelen = 0;
+
+	if (smc->sk.sk_state == SMC_INIT)
+		return -ENOTCONN;
+
+	buffer_len = conn->rmb_rx_size;
+	buffer =  conn->rmb_rx_elem->rmb_rx;
+	len = min_t(size_t, len, conn->rmb_rx_size);
+
+	if (flags & MSG_OOB)
+		return smc_conn_recv_urg(smc, msg, len, flags);
+
+again:
+	target = sock_rcvlowat(&smc->sk, flags & MSG_WAITALL, len);
+	timeo = sock_rcvtimeo(&smc->sk, flags & MSG_DONTWAIT);
+	if (signal_pending(current)) {
+		rc = timeo ? -EINTR : -EAGAIN;
+		return rc;
+	}
+	rc = smc_wait_rx_data(smc, target, flags & MSG_WAITALL, timeo);
+	if ((rc == -EAGAIN) || (rc == -EINTR))
+		return rc;
+	if (!rc) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			release_sock(&smc->sk);
+			smc_terminate_conn(smc->conn.lgr);
+			lock_sock(&smc->sk);
+			return -EINTR;
+		}
+		goto again;
+	}
+	if (smc_splice_in_use(smc)) {
+		if (smc_splice_in_progress(smc) && (flags &  MSG_DONTWAIT))
+			return -EAGAIN;
+		wait_event_interruptible(smc->splice_waiter,
+					 !smc_splice_in_progress(smc));
+	}
+	toread = smc_to_read(smc);
+	rc = 0;
+
+	if ((toread <= 0) &&
+	    (smc->sk.sk_err ||
+	     smc->sk.sk_shutdown & RCV_SHUTDOWN ||
+	     sock_flag(&smc->sk, SOCK_DONE) ||
+	     smc_stop_received(conn)))
+		return sock_error(&smc->sk);
+
+	prod.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	if (toread <= 0)
+		goto check_repeat;
+
+	if ((conn->rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+		if (smc_curs_diff(conn->rmb_rx_size, &cons, &urg))
+			prod = urg;
+		else /* data preceding urg already received */
+			conn->rx_urg_state |= SMC_URG_RECV;
+	} else {
+		urg.s.lcurs = 0;
+	}
+
+	if ((toread < target) && !smc_stop_received(conn))
+		goto check_repeat;
+
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left = prod.s.curs.c - cons.s.curs.c;
+		copy = min_t(size_t, len, space_left);
+		if (!(flags & MSG_TRUNC)) {
+			if (memcpy_to_msg(msg, buffer + cons.s.curs.c, copy))
+				return -EFAULT;
+		}
+		len -= copy;
+		rc += copy;
+	} else {
+		space_left = buffer_len - cons.s.curs.c;
+		if (space_left) {
+			copy = min_t(size_t, len, space_left);
+			if (!(flags & MSG_TRUNC)) {
+				if (memcpy_to_msg(msg, buffer + cons.s.curs.c,
+						  copy))
+					return -EFAULT;
+			}
+			len -= copy;
+			rc += copy;
+		}
+		if (len) {
+			space_left = prod.s.curs.c;
+			copy = min_t(size_t, len, space_left);
+			if (!(flags & MSG_TRUNC)) {
+				if (memcpy_to_msg(msg, buffer, copy))
+					return -EFAULT;
+			}
+			len -= copy;
+			rc += copy;
+		}
+	}
+
+	/* update cursors */
+	if (!(flags & MSG_PEEK)) {
+		smc_curs_add(buffer_len, &cons.s.curs, rc);
+		smc_skip_urg_data(smc, &cons, &urg);
+		smp_mb__before_atomic();
+		atomic_sub(rc, &conn->bytes_to_rcv);
+		smp_mb__after_atomic();
+		atomic64_set(&conn->local_tx_ctrl.c_curs.s.acurs, cons.s.lcurs);
+		spl_cons.s.lcurs =
+			atomic64_read(&conn->local_rx_spliced.s.acurs);
+		if (spl_cons.s.curs.reserved0)
+			atomic64_set(&conn->local_rx_spliced.s.acurs,
+				     cons.s.lcurs);
+		/* send consumer cursor update if required */
+		smc_update_cons_curs(smc, NULL);
+	}
+check_repeat:
+	if ((toread < target) &&
+	    !smc_stop_received(conn) &&
+	    !conn->local_tx_ctrl.conn_state_flags.abnormal_close) {
+		loop = 0;
+		goto again;
+	}
+
+	return rc;
+}
+
+ssize_t smc_conn_splice_read(struct smc_sock *smc, struct pipe_inode_info *pipe,
+			     size_t len, unsigned int flags)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0, ret = 0;
+	int toread = 0, space_left, copy, buffer_len;
+	int target = 1;		/* read at least these many bytes */
+	char *buffer;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs spl_cons;
+	struct smc_curs urg;
+	long timeo;
+	int loop = 0;
+
+	if (smc->sk.sk_state == SMC_INIT)
+		return -ENOTCONN;
+
+	buffer_len = conn->rmb_rx_size;
+	buffer =  conn->rmb_rx_elem->rmb_rx;
+	len = min_t(size_t, len, conn->rmb_rx_size);
+
+again:
+	timeo = sock_rcvtimeo(&smc->sk, 0);
+	if (signal_pending(current)) {
+		rc = timeo ? -EINTR : -EAGAIN;
+		return rc;
+	}
+	rc = smc_wait_rx_data(smc, target, 0, timeo);
+	if ((rc == -EAGAIN) || (rc == -EINTR))
+		return rc;
+	if (!rc) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			release_sock(&smc->sk);
+			smc_terminate_conn(smc->conn.lgr);
+			lock_sock(&smc->sk);
+			return -EINTR;
+		}
+		goto again;
+	}
+	toread = smc_to_read(smc);
+
+	if ((toread <= 0) &&
+	    (smc->sk.sk_err ||
+	     smc->sk.sk_shutdown & RCV_SHUTDOWN ||
+	     sock_flag(&smc->sk, SOCK_DONE) ||
+	     smc_stop_received(conn)))
+		return sock_error(&smc->sk);
+
+	rc = 0;
+	prod.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	if (toread <= 0)
+		goto check_repeat;
+
+	if ((conn->rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+		if (smc_curs_diff(conn->rmb_rx_size, &cons, &urg))
+			prod = urg;
+		else /* data preceding urg already received */
+			conn->rx_urg_state |= SMC_URG_RECV;
+	} else {
+		urg.s.lcurs = 0;
+	}
+
+	if ((toread < target) && !smc_stop_received(conn))
+		goto check_repeat;
+
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left = prod.s.curs.c - cons.s.curs.c;
+		copy = min_t(size_t, len, space_left);
+		rc = smc_splice_to_pipe(smc, pipe, flags,
+					buffer + cons.s.curs.c, copy);
+		if (rc < 0)
+			return -EFAULT;
+		len -= rc;
+	} else {
+		space_left = buffer_len - cons.s.curs.c;
+		if (space_left) {
+			copy = min_t(size_t, len, space_left);
+			rc = smc_splice_to_pipe(smc, pipe, flags,
+						buffer + cons.s.curs.c,
+						copy);
+			if (rc < 0)
+				return -EFAULT;
+			len -= rc;
+		}
+		if (len) {
+			space_left = prod.s.curs.c;
+			copy = min_t(size_t, len, space_left);
+			ret = smc_splice_to_pipe(smc, pipe, flags, buffer,
+						 copy);
+			if (ret < 0)
+				return -EFAULT;
+			len -= ret;
+			rc += ret;
+		}
+	}
+
+	/* update cursors */
+	spl_cons.s.lcurs = atomic64_read(&conn->local_rx_spliced.s.acurs);
+	if (!spl_cons.s.curs.reserved0) {
+		/* no previous splice in progress, initialize to current */
+		spl_cons = cons;
+		spl_cons.s.curs.reserved0 = 1;
+	}
+	smc_curs_add(buffer_len, &spl_cons.s.curs, rc);
+	smc_skip_urg_data(smc, &spl_cons, &urg);
+	atomic64_set(&conn->local_rx_spliced.s.acurs, spl_cons.s.lcurs);
+	/* we update cursors in a callback after splicing completed */
+check_repeat:
+	if ((toread < target) &&
+	    !smc_stop_received(conn) &&
+	    !conn->local_tx_ctrl.conn_state_flags.abnormal_close) {
+		loop = 0;
+		goto again;
+	}
+
+	return rc;
+}
+
+void smc_check_qp_attr(struct smc_link *lnk)
+{
+	int rc;
+	struct ib_qp_attr *attr = &lnk->qp_attr;
+	struct ib_qp_init_attr init_attr;
+
+	memset(attr, 0, sizeof(struct ib_qp_attr));
+	memset(&init_attr, 0, sizeof(struct ib_qp_init_attr));
+	rc = ib_query_qp(lnk->roce_qp, attr,
+			 IB_QP_STATE |
+			 IB_QP_CUR_STATE |
+			 IB_QP_PKEY_INDEX |
+			 IB_QP_PORT |
+			 IB_QP_QKEY |
+			 IB_QP_AV |
+			 IB_QP_PATH_MTU |
+			 IB_QP_TIMEOUT |
+			 IB_QP_RETRY_CNT |
+			 IB_QP_RNR_RETRY |
+			 IB_QP_RQ_PSN |
+			 IB_QP_ALT_PATH |
+			 IB_QP_MIN_RNR_TIMER |
+			 IB_QP_SQ_PSN |
+			 IB_QP_PATH_MIG_STATE |
+			 IB_QP_CAP |
+			 IB_QP_VID |
+			 IB_QP_DEST_QPN,
+			 &init_attr);
+
+	lnk->send_wr_num = min_t(size_t, smc_ctrl_buf_cnt,
+				 lnk->qp_attr.cap.max_send_wr);
+	lnk->recv_wr_num = min_t(size_t, smc_ctrl_buf_cnt * 10,
+				 lnk->qp_attr.cap.max_recv_wr);
+}
+
+void smc_check_port_attr(struct smc_ib_device *smc_ibdev, int port_num)
+{
+	struct ib_device *dev = smc_ibdev->dev;
+	struct ib_port_attr *pattr;
+	int rc;
+
+	pattr = &smc_ibdev->pattr[port_num];
+	memset(pattr, 0, sizeof(struct ib_port_attr));
+	rc = ib_query_port(dev, port_num + 1, pattr);
+}
+
+void smc_check_dev_attr(struct smc_ib_device *smc_ibdev)
+{
+	struct ib_device *dev = smc_ibdev->dev;
+	struct ib_device_attr *attr = &smc_ibdev->attr;
+	int i, rc;
+
+	memset(attr, 0, sizeof(struct ib_device_attr));
+	rc = ib_query_device(dev, attr);
+
+	for (i = 0; i < 2; i++)
+		smc_check_port_attr(smc_ibdev, i);
+}
+
diff --git a/net/smc/smc_llc.c b/net/smc/smc_llc.c
new file mode 100644
index 0000000..7f6eb2a
--- /dev/null
+++ b/net/smc/smc_llc.c
@@ -0,0 +1,1597 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  SMC-R Link Layer Control
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Klaus Wacker <Klaus.Wacker@de.ibm.com>
+ *              Ursula Braun <ursula.braun@de.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/wait.h>
+#include <linux/kthread.h>
+#include <linux/hardirq.h>
+#include <linux/types.h>
+#include <net/tcp.h>
+#include <net/sock.h>
+#ifdef CONFIG_S390
+#include <asm/debug.h>
+#endif
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <linux/splice.h>
+#include <linux/mm.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+static enum llc_msg_group llc_msg_to_grp(enum llc_msg_type type)
+{
+	enum llc_msg_group grp;
+
+	switch (type) {
+	case LLC_ADD_LINK:
+		grp = LLC_GRP_ADD_LINK;
+		break;
+	case LLC_DEL_LINK:
+		grp = LLC_GRP_DEL_LINK;
+		break;
+	case LLC_CONFIRM_RKEY:
+		grp = LLC_GRP_CONF_RKEY;
+		break;
+	case LLC_TEST_LINK:
+		grp = LLC_GRP_TEST_LINK;
+		break;
+	default:
+		grp = LLC_GRP_NONE;
+	}
+	return grp;
+}
+
+int llc_enqueue(struct smc_link *lnk, struct llc_msg *msg)
+{
+	struct smc_link_group *lgr;
+	struct llc_qentry *qentry;
+	unsigned long flags;
+
+	lgr = lnk->lnk_grp;
+	qentry = kzalloc(sizeof(*qentry), GFP_ATOMIC);
+	if (!qentry)
+		return -ENOMEM;
+	qentry->link = lnk;
+	memcpy(&qentry->msg, msg, sizeof(struct llc_msg));
+	if (msg->hd.flags & LLC_FLAG_RESP) {
+		switch (msg->hd.type) {
+		case LLC_ADD_LINK:
+		case LLC_DEL_LINK:
+		case LLC_CONFIRM_LINK:
+		case LLC_ADD_LINK_CONT:
+			lgr->llc_ctl[LLC_SERV].qentry = qentry;
+			break;
+		case LLC_CONFIRM_RKEY:
+		case LLC_CONFIRM_RKEY_CONT:
+		case LLC_DELETE_RKEY:
+			lgr->llc_ctl[lgr->role].qentry = qentry;
+			break;
+		case LLC_TEST_LINK:
+			lgr->llc_ctl[LLC_TESTLINK].qentry = qentry;
+			break;
+		default:
+			llc_send_del_link(lnk, NULL, 0, LLC_DEL_PROT_VIOL,
+					  LLC_NO_NOTIFY);
+			smc_terminate_conn(lgr);
+			kfree(qentry);
+			break;
+		}
+		wake_up_interruptible(&lgr->llc_waiter);
+		return 0;
+	}
+	spin_lock_irqsave(&lgr->llc_event_q_lock, flags);
+	list_add_tail(&qentry->list, &lgr->llc_event_q);
+	spin_unlock_irqrestore(&lgr->llc_event_q_lock, flags);
+	if (lgr->llc_wq)
+		queue_work(lgr->llc_wq, &lgr->llc_event_work);
+
+	return 0;
+}
+
+int llc_initiate(struct smc_link_group *lgr, enum llc_msg_group llc_grp)
+{
+	enum llc_msg_group llc_grp_allowed;
+	int peer;
+	enum llc_ctl_ind llc_seq;
+	int rest_time = 1;
+
+	llc_grp_allowed = (llc_grp == LLC_GRP_CONF_RKEY) ?
+				LLC_GRP_CONF_RKEY : LLC_GRP_NONE;
+	peer = !lgr->role;
+	switch (llc_grp) {
+	case LLC_GRP_CONF_RKEY:
+		llc_seq = (enum llc_ctl_ind)lgr->role;
+		break;
+	case LLC_GRP_TEST_LINK:
+		llc_seq = 2;
+		peer = 2;
+		llc_grp_allowed = LLC_GRP_TEST_LINK;
+		break;
+	default:
+		llc_seq = LLC_SERV;
+	}
+reserve_llc:
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	if ((lgr->llc_ctl[llc_seq].active == LLC_GRP_NONE) &&
+	    (lgr->llc_ctl[peer].active <= llc_grp_allowed)) {
+		lgr->llc_ctl[llc_seq].active = llc_grp;
+		spin_unlock_bh(&lgr->llc_ctl_lock);
+		return rest_time;
+	}
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	rest_time = wait_event_interruptible_timeout(lgr->llc_waiter,
+		((lgr->llc_ctl[llc_seq].active == LLC_GRP_NONE) &&
+		 (lgr->llc_ctl[peer].active <= llc_grp_allowed)),
+		LLC_WAIT_TIMEO);
+	if (!rest_time)
+		return rest_time;
+	goto reserve_llc;
+}
+
+void llc_stop(struct smc_link_group *lgr, int llc_grp)
+{
+	enum llc_ctl_ind llc_seq;
+
+	switch (llc_grp) {
+	case LLC_GRP_CONF_RKEY:
+		llc_seq = (enum llc_ctl_ind)lgr->role;
+		break;
+	case LLC_GRP_TEST_LINK:
+		llc_seq = 2;
+		break;
+	default:
+		llc_seq = LLC_SERV;
+	}
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	memset(&lgr->llc_ctl[llc_seq], 0, sizeof(struct llc_ctl));
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	if (lgr->delayed_q &&
+	    (llc_grp != LLC_GRP_TEST_LINK)) {
+		queue_work(lgr->llc_wq, &lgr->llc_event_work);
+	} else {
+		wake_up_interruptible(&lgr->llc_waiter);
+	}
+}
+
+struct llc_qentry *llc_wait(struct smc_link_group *lgr, enum smc_role initiator,
+			    int time_out, u8 exp_msg)
+{
+	struct llc_ctl *ctl = &lgr->llc_ctl[initiator];
+
+	wait_event_interruptible_timeout(lgr->llc_waiter,
+					 (ctl->qentry ||
+					 ((ctl->ctrl_elem == LLC_SEND_POSTED) &&
+					  ctl->wc_status) ||
+					 (lgr->lgr_type == NONE)),
+					 time_out);
+	if (IS_ERR(ctl->qentry) ||	/* waiter has been killed */
+	    !ctl->qentry) {
+		ctl->qentry = NULL;
+		goto out;
+	}
+	if (((initiator == SMC_SERV) ||
+	     (exp_msg != LLC_CONFIRM_LINK)) &&
+	    (ctl->ctrl_elem != LLC_SEND_POSTED)) {
+		/* cancel post req when send completes after llc msg arrival */
+		lgr->lnk[ctl->lnk_idx].pending[ctl->ctrl_elem].post_ctl =
+			LLC_NO_NOTIFY; /* mark as cancelled */
+	}
+	if ((exp_msg) && (ctl->qentry->msg.hd.type != exp_msg)) {
+		if ((exp_msg == LLC_ADD_LINK) &&
+		    (ctl->qentry->msg.hd.type == LLC_DEL_LINK)) {
+			schedule_work(&lgr->llc_del_link_work);
+			return NULL;
+		}
+		kfree(ctl->qentry);
+		ctl->qentry = NULL;
+	}
+out:
+	return ctl->qentry;
+}
+
+static int llc_add_pending_send(struct smc_link *link,
+				enum llc_ctl_ind ctl_idx, u32 *elem)
+{
+	u64 wr_id;
+	int rc = 0;
+
+	wr_id = (u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	rc = smc_get_ctrl_buf(link, elem);
+	if (rc < 0)
+		return rc;
+	memset(&link->pending[*elem], 0, sizeof(struct pending_send));
+	link->pending[*elem].wr_id_send = wr_id;
+	link->pending[*elem].post_ctl = ctl_idx;
+	if (ctl_idx != LLC_NO_NOTIFY)
+		link->lnk_grp->llc_ctl[ctl_idx].ctrl_elem = *elem;
+	link->send_wr[*elem].wr_id = wr_id;
+	return 0;
+}
+
+static inline int llc_send_msg(struct smc_link *link, struct llc_msg *msg,
+			       enum llc_ctl_ind ctl_idx)
+{
+	struct smc_link_group *lgr;
+	u32 index = link->send_wr_num;
+	int rc;
+
+	if ((atomic_read(&link->state) != SMC_LINK_UP) || !link->lnk_grp)
+		return -EPIPE;
+	lgr = link->lnk_grp;
+	if (lgr->lgr_type == NONE)
+		return -EPIPE;
+	if (ctl_idx != LLC_NO_NOTIFY)
+		lgr->llc_ctl[ctl_idx].lnk_idx = link - &lgr->lnk[0];
+	/* send llc message */
+	rc = llc_add_pending_send(link, ctl_idx, &index);
+	if (!rc)
+		rc = smc_wr_send(link, NULL, (u64)msg, index);
+	if (rc) {
+		if (index < link->send_wr_num)
+			test_and_clear_bit(index, link->pending_bit_mask);
+	}
+	return rc;
+}
+
+int llc_send_confirm_link(struct smc_link *link, u8 mac[], union ib_gid *gid,
+			  enum llc_reqresp reqresp, enum llc_ctl_ind ctl_idx)
+{
+	struct llc_confirm_msg confllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	u32 index;
+	int rc = 0;
+
+	memset(&confllc, 0, sizeof(confllc));
+	confllc.hd.type = LLC_CONFIRM_LINK;
+	confllc.hd.length = sizeof(struct llc_confirm_msg);
+	if (reqresp == LLC_RESP)
+		confllc.hd.flags = 0x80;
+	else
+		confllc.hd.flags = 0x00;
+	confllc.hd.flags |= SMC_CACHE_ALIGN_SIZE_SHORT;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	memcpy(confllc.sender_mac, mac, 6);
+	memcpy(confllc.sender_gid, gid, 16);
+	confllc.max_links = lgr->max_links;
+	confllc.link_num = link->link_id;
+	confllc.link_uid = htonl(lgr->lgr_id + link->link_id);
+
+	hton_three(link->roce_qp->qp_num, confllc.sender_qp_num);
+	/* send llc message */
+	if (ctl_idx != LLC_NO_NOTIFY)
+		lgr->llc_ctl[ctl_idx].lnk_idx = link - &lgr->lnk[0];
+	rc = llc_add_pending_send(link, ctl_idx, &index);
+	if (!rc)
+		rc = smc_wr_send(link, NULL, (u64)&confllc, index);
+	return rc;
+}
+
+int llc_send_add_link(struct smc_link *link, struct smc_roce_defs *rocdefs,
+		      struct smc_link *linkw, enum llc_reqresp reqresp,
+		      enum llc_ctl_ind ctl_idx)
+{
+	struct llc_add_link_msg addllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int rc = 0;
+
+	memset(&addllc, 0, sizeof(addllc));
+	addllc.hd.type = LLC_ADD_LINK;
+	addllc.hd.length = sizeof(struct llc_add_link_msg);
+	if (reqresp == LLC_RESP) {
+		addllc.hd.flags = 0x80;
+		addllc.flags2 = min(linkw->path_mtu, linkw->mtu_peer);
+	} else {
+		addllc.hd.flags = 0x00;
+		if (linkw)
+			addllc.flags2 = linkw->path_mtu;
+		else
+			addllc.flags2 = link->path_mtu;
+		if (lgr->role == SMC_SERV)
+			lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	}
+	memcpy(addllc.sender_mac, &rocdefs->mac, 6);
+	memcpy(addllc.sender_gid, &rocdefs->gid, 16);
+	if (linkw) {
+		addllc.link_num = linkw->link_id;
+		hton_three(linkw->roce_qp->qp_num, addllc.sender_qp_num);
+		hton_three(linkw->psn_initial, addllc.initial_psn);
+	}
+	/* send llc message */
+	rc = llc_send_msg(link, (struct llc_msg *)&addllc, ctl_idx);
+	return rc;
+}
+
+int llc_send_del_link(struct smc_link *link, struct smc_link *link2,
+		      u8 ord_flag, u32 reason, enum llc_ctl_ind ctl_idx)
+{
+	struct llc_del_link_msg delllc;
+
+	memset(&delllc, 0, sizeof(delllc));
+	delllc.hd.type = LLC_DEL_LINK;
+	delllc.hd.length = sizeof(struct llc_del_link_msg);
+	delllc.hd.flags = ord_flag;
+	if (link2) {
+		delllc.link_num = link2->link_id;
+	} else {
+		delllc.link_num = 0;
+		delllc.hd.flags |= LLC_FLAG_DEL_ALL;
+	}
+	delllc.reason = htonl(reason);
+
+	return llc_send_msg(link, (struct llc_msg *)&delllc, ctl_idx);
+}
+
+int llc_do_confirm_rkey(struct smc_link *link, struct rmb_rx_addrs *rmb_rx_elem)
+{
+	struct llc_confirm_rkey_msg confrkeyllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int prim_link_id, rt_idx, i, prim_lnk_idx;
+
+	memset(&confrkeyllc, 0, sizeof(confrkeyllc));
+	confrkeyllc.hd.type = LLC_CONFIRM_RKEY;
+	confrkeyllc.hd.length = sizeof(struct llc_confirm_rkey_msg);
+	confrkeyllc.hd.flags = 0x00;
+	lgr->llc_ctl[lgr->role].qentry = NULL;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	confrkeyllc.rt[0].rmb_key =
+		htonl(rmb_rx_elem->mr_rx[prim_lnk_idx]->rkey);
+	confrkeyllc.rt[0].rmb_vaddr =
+		cpu_to_be64((u64)rmb_rx_elem->rmb_rx_dma[prim_lnk_idx]);
+
+	prim_link_id = link->link_id;
+	rt_idx = 1;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((lgr->lnk[i].link_id == prim_link_id) ||
+		    (atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP))
+			continue;
+		confrkeyllc.rt[rt_idx].link_id = lgr->lnk[i].link_id;
+		confrkeyllc.rt[rt_idx].rmb_key =
+			htonl(rmb_rx_elem->mr_rx[i]->rkey);
+		confrkeyllc.rt[rt_idx].rmb_vaddr =
+			cpu_to_be64((u64)rmb_rx_elem->rmb_rx_dma[i]);
+		rt_idx = rt_idx + 1;
+	}
+	confrkeyllc.rt[0].link_id = rt_idx - 1;
+	if (rt_idx == 1) /* nothing to confirm */
+		WARN_ON(1);
+
+	return llc_send_msg(link, (struct llc_msg *)&confrkeyllc,
+			    (enum llc_ctl_ind)lgr->role);
+}
+
+int llc_do_del_rkey(struct smc_link *link, struct rmb_rx_addrs *rmb_rx_elem)
+{
+	struct llc_delete_rkey_msg delrkeyllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int prim_lnk_idx;
+
+	memset(&delrkeyllc, 0, sizeof(delrkeyllc));
+	delrkeyllc.hd.type = LLC_DELETE_RKEY;
+	delrkeyllc.hd.length = sizeof(struct llc_delete_rkey_msg);
+	delrkeyllc.hd.flags = 0x00;
+	lgr->llc_ctl[lgr->role].qentry = NULL;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	delrkeyllc.num_rkeys = 1;
+	delrkeyllc.rk[0] =
+		htonl(rmb_rx_elem->mr_rx[prim_lnk_idx]->rkey);
+
+	return llc_send_msg(link, (struct llc_msg *)&delrkeyllc,
+			    (enum llc_ctl_ind)lgr->role);
+}
+
+struct bufs_list_pos {
+	struct list_head	*head;
+	int			ind;
+	struct rmb_rx_addrs	*elem;
+};
+
+static void llc_get_next_rx_rmb_slot(struct smc_link_group *lgr,
+				     struct bufs_list_pos *bufs_pos)
+{
+	int start;
+
+	start = bufs_pos->ind;
+	if (bufs_pos->elem && bufs_pos->elem->list.next !=
+		&bufs_pos->head[start]) {
+		bufs_pos->elem =
+			(struct rmb_rx_addrs *)bufs_pos->elem->list.next;
+		return;
+	}
+	while (start < (SMC_RMBE_SIZE_VALUES - 1)) {
+		start++;
+		if (bufs_pos->head[start].next !=
+			&bufs_pos->head[start]) {
+			bufs_pos->elem = (struct rmb_rx_addrs *)
+						bufs_pos->head[start].next;
+			bufs_pos->ind = start;
+			return;
+		}
+	}
+}
+
+static int llc_add_link_cont(struct smc_link *link, int lnk_idx, u8 link_num,
+			     u8 *num_rkeys_todo, struct bufs_list_pos *pos)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_add_link_cont_msg addc_llcs;
+	int prim_lnk_idx, i;
+	u8 n;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	addc_llcs.link_num = link_num;
+	addc_llcs.num_rkeys = *num_rkeys_todo;
+	n = *num_rkeys_todo;
+	memset(&addc_llcs.reserved2, 0,
+	       sizeof(struct llc_add_link_cont_msg) -
+		      offsetof(struct llc_add_link_cont_msg, reserved2));
+	for (i = 0; i < min_t(u8, n, 2); i++) {
+next_rmb_slot:
+		llc_get_next_rx_rmb_slot(lgr, pos);
+		if (!pos->elem) {
+			addc_llcs.num_rkeys = addc_llcs.num_rkeys -
+							*num_rkeys_todo;
+			*num_rkeys_todo = 0;
+			break;
+		}
+		if (!pos->elem->mr_rx[prim_lnk_idx])
+			goto next_rmb_slot;
+		addc_llcs.rt[i].rmb_key =
+			htonl(pos->elem->mr_rx[prim_lnk_idx]->rkey);
+		addc_llcs.rt[i].rmb_key_new =
+			htonl(pos->elem->mr_rx[lnk_idx]->rkey);
+		addc_llcs.rt[i].rmb_vaddr_new =
+			cpu_to_be64((u64)pos->elem->rmb_rx_dma[lnk_idx]);
+		(*num_rkeys_todo)--;
+	}
+	addc_llcs.hd.type = LLC_ADD_LINK_CONT;
+	addc_llcs.hd.length = sizeof(struct llc_msg);
+	addc_llcs.hd.version = SMC_CLC_V1;
+	if (lgr->role == SMC_SERV)
+		addc_llcs.hd.flags = 0x00;
+	else
+		addc_llcs.hd.flags = 0x80;
+	llc_send_msg(link, (struct llc_msg *)&addc_llcs, LLC_SERV);
+	return 0;
+}
+
+int llc_send_test_link(struct smc_link *link, u8 udata[])
+{
+	struct llc_test_link_msg testllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+
+	testllc.hd.type = LLC_TEST_LINK;
+	memset(&testllc.user_data, 0, sizeof(testllc.user_data));
+	strncpy(testllc.user_data, udata, sizeof(testllc.user_data));
+	testllc.hd.length = sizeof(struct llc_test_link_msg);
+	testllc.hd.flags = 0x00;
+	lgr->llc_ctl[LLC_TESTLINK].qentry = NULL;
+	testllc.hd.version = SMC_CLC_V1;
+	return llc_send_msg(link, (struct llc_msg *)&testllc, LLC_TESTLINK);
+}
+
+int llc_get_fail_cause(struct smc_link_group *lgr, enum llc_ctl_ind initiator,
+		       struct smc_sock *smc)
+{
+	struct smc_decline_clc_msg dclc;
+	int reason_code, rc;
+
+	if ((initiator < 3) &&
+	    ((lgr->llc_ctl[initiator].ctrl_elem != LLC_SEND_POSTED) ||
+	     !lgr->llc_ctl[initiator].wc_status)) {
+		/* llc send failure */
+		return -ECOMM;
+	}
+	if (!smc)
+		return -ETIMEDOUT;
+	/* clc decl received or llc timeout */
+	memset(&dclc, 0, sizeof(dclc));
+	rc = clc_wait_msg(smc, SMC_CLC_NONBLOCK, (char *)&dclc,
+			  sizeof(struct smc_decline_clc_msg),
+			  SMC_CLC_DECLINE, &reason_code);
+	/* clc decline received */
+	return rc;
+}
+
+static u8 llc_count_rx_rmb_slot(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct rmb_rx_addrs *entry_rx;
+	int i;
+	u8 count;
+
+	count = 0;
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(entry_rx, &lgr->rmb_rx_bufs[i], list) {
+			if (entry_rx->mr_rx[lnk_idx])
+				count++;
+		}
+	}
+	return count;
+}
+
+int smc_find_rtoken_by_link(struct smc_link_group *lgr, int lnk_idx, u32 rkey)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		if ((lgr->rtok[i].rkey[lnk_idx] == rkey) &&
+		    (lgr->rtok[i].link_id[lnk_idx] ==
+						lgr->lnk[lnk_idx].link_id) &&
+		    (atomic_read(&lgr->rtok[i].active) == 1))
+			return i;
+	}
+	return -ENOENT;
+}
+
+static void llc_set_rtoken(struct smc_link_group *lgr, int link_n1, int link_n2,
+			   struct rmb_rtoken2 *rtoken)
+{
+	u32 rkey;
+	int rtok_idx;
+
+	rkey = ntohl(rtoken->rmb_key);
+	rtok_idx = smc_find_rtoken_by_link(lgr, link_n1, rkey);
+	if (rtok_idx == -ENOENT)
+		return;
+	lgr->rtok[rtok_idx].rkey[link_n2] = ntohl(rtoken->rmb_key_new);
+	lgr->rtok[rtok_idx].vaddr[link_n2] = be64_to_cpu(rtoken->rmb_vaddr_new);
+	lgr->rtok[rtok_idx].link_id[link_n2] = lgr->lnk[link_n2].link_id;
+}
+
+int smc_get_rtoken(struct smc_link_group *lgr)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		if (atomic_cmpxchg(&lgr->rtok[i].active, 0, 1) == 0)
+			return i;
+	}
+	return -ENOSPC;
+}
+
+static void llc_set_rtoken_by_link(struct smc_link_group *lgr, int link_n1,
+				   int rtoken_idx, struct rmb_rtoken3 *rtoken)
+{
+	int lnk_idx, found = 0;
+
+	for (lnk_idx = 0; lnk_idx <= SMC_MAX_SYM_LINKS; lnk_idx++) {
+		if (lgr->lnk[lnk_idx].link_id == link_n1) {
+			found = 1;
+			break;
+		}
+	}
+	if (!found) {
+		WARN_ON(1);
+		return;
+	}
+	lgr->rtok[rtoken_idx].rkey[lnk_idx] = ntohl(rtoken->rmb_key);
+	lgr->rtok[rtoken_idx].vaddr[lnk_idx] = be64_to_cpu(rtoken->rmb_vaddr);
+	lgr->rtok[rtoken_idx].link_id[lnk_idx] = link_n1;
+}
+
+static int smc_clear_rtokens_by_link(struct smc_link_group *lgr, int link_n1,
+				     __be32 rmb_rkey)
+{
+	int i, lnk_idx, rtoken_idx, found = 0;
+	u32 rkey;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id == link_n1) {
+			found = 1;
+			break;
+		}
+	}
+	if (!found)
+		return -ENODEV;
+	lnk_idx = i;
+	rkey = ntohl(rmb_rkey);
+	rtoken_idx = smc_find_rtoken_by_link(lgr, lnk_idx, rkey);
+	found = (rtoken_idx < 0 ? 0 : 1);
+	if (!found)
+		return -ENODEV;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		lgr->rtok[rtoken_idx].rkey[i] = 0;
+		lgr->rtok[rtoken_idx].vaddr[i] = 0;
+		lgr->rtok[rtoken_idx].link_id[i] = 0;
+	}
+	atomic_set(&lgr->rtok[rtoken_idx].active, 0);
+	return 0;
+}
+
+static int llc_alloc_alt_link(struct smc_link_group *lgr,
+			      enum lgr_type lgr_new_t)
+{
+	int i, lnk_idx, total_links = 0;
+
+	if ((lgr->lgr_type != SINGLE) &&
+	    ((lgr_new_t == ASYMMETRIC_LOCAL) ||
+	     (lgr_new_t == ASYMMETRIC_PEER)))
+		return -EMLINK;
+	if ((lgr_new_t == ASYMMETRIC_LOCAL) ||
+	    (lgr_new_t == ASYMMETRIC_PEER)) {
+		lnk_idx = SMC_MAX_SYM_LINKS;
+		for (i = SMC_MAX_SYM_LINKS; i > 0; i--) {
+			if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_FREED) {
+				lnk_idx = i;
+				break;
+			}
+		}
+	} else {
+		lnk_idx = SMC_MAX_SYM_LINKS + 1;
+		total_links = 0;
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_FREED) {
+				if (lnk_idx > SMC_MAX_SYM_LINKS)
+					lnk_idx = i;
+			} else {
+				total_links++;
+			}
+		}
+		if ((lgr->lgr_type == SYMMETRIC) &&
+		    (total_links >= SMC_MAX_SYM_LINKS))
+			return -EMLINK;
+		if (lnk_idx > SMC_MAX_SYM_LINKS)
+			return -EMLINK;
+	}
+	return lnk_idx;
+}
+
+static void llc_deny_req(struct llc_qentry *qentry, u8 flag, u8 retc,
+			 enum llc_ctl_ind ctl_idx)
+{
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	qentry->msg.hd.flags |= flag;
+	qentry->msg.hd.version |= retc;
+	llc_send_msg(qentry->link, &qentry->msg, ctl_idx);
+	kfree(qentry);
+}
+
+static void llc_save_add_link_info(struct smc_link *link,
+				   struct llc_add_link_msg *add_llc)
+{
+	memcpy(link->gid_peer, ((union ib_gid *)add_llc->sender_gid)->raw, 16);
+	memcpy(link->mac_peer, add_llc->sender_mac, 6);
+	ntoh_three(&link->qp_num_peer, add_llc->sender_qp_num);
+	ntoh_three(&link->psn_peer, add_llc->initial_psn);
+	link->mtu_peer = add_llc->flags2 & 0x0f;
+}
+
+static int llc_cli_rkey_exchange(struct smc_link *link,
+				 struct smc_link *link_new,
+				 struct smc_sock *smc, int lnk_idx)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_cont_msg *addc_llc;
+	struct bufs_list_pos pos = {
+		.head = lgr->rmb_rx_bufs,
+		.ind = -1,
+		.elem = NULL,
+	};
+	int prim_lnk_idx, i;
+	u8 n, num_rkeys_send, num_rkeys_recv;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	num_rkeys_send = llc_count_rx_rmb_slot(lgr, prim_lnk_idx);
+	do {
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK_CONT);
+		if (!qentry) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return llc_get_fail_cause(lgr, LLC_SERV, smc);
+		}
+		addc_llc = (struct llc_add_link_cont_msg *)&qentry->msg;
+		num_rkeys_recv = addc_llc->num_rkeys;
+		n = addc_llc->num_rkeys;
+		for (i = 0; i < min_t(u8, n, 2); i++) {
+			llc_set_rtoken(lgr, prim_lnk_idx, lnk_idx,
+				       &addc_llc->rt[i]);
+			num_rkeys_recv--;
+		}
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		kfree(qentry);
+		llc_add_link_cont(link, lnk_idx, link_new->link_id,
+				  &num_rkeys_send, &pos);
+	} while (num_rkeys_send || num_rkeys_recv);
+	return 0;
+}
+
+static int llc_cli_conf_link(struct smc_link *link,
+			     struct smc_roce_defs *rocedef,
+			     struct smc_link *link_new, int lnk_idx,
+			     enum lgr_type lgr_new_t)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_del_link_msg *del_llc;
+	int rc = 0, cas;
+
+	/* receive CONFIRM LINK request over RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO, 0);
+	if (!qentry) {
+		rc = llc_send_del_link(link, link_new, 0,
+				       LLC_DEL_LOST_PATH,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	if (qentry->msg.hd.type == LLC_CONFIRM_LINK) {
+		rc = smc_modify_qp_rts(link_new);
+		cas = 0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+		if (rc || !(qentry->msg.hd.flags & 0xf) ||
+		    !lgr->cache_align_size || (lgr->cache_align_size != cas)) {
+			/* no cache alignment specified, decline */
+			kfree(qentry);
+			rc = llc_send_del_link(link, link_new, 0,
+					       LLC_DEL_PROT_VIOL,
+					       LLC_NO_NOTIFY);
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return -ENOLINK;
+		}
+		if (!lgr->cache_align_size)
+			lgr->cache_align_size = cas;
+		smc_check_qp_attr(link_new);
+		/* send CONFIRM LINK response over RoCE fabric */
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		rc = llc_send_confirm_link(link_new,
+					   (u8 *)&rocedef->mac,
+					   &rocedef->gid, LLC_RESP,
+					   LLC_NO_NOTIFY);
+		kfree(qentry);
+		if (rc) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			rc = llc_send_del_link(link, link_new, 0,
+					       LLC_DEL_LOST_PATH,
+					       LLC_NO_NOTIFY);
+			if (rc && smc_lnk_downing(&link->state))
+				smc_link_down(lgr, link - &lgr->lnk[0]);
+			return -ENOLINK;
+		}
+		atomic_set(&link_new->state, SMC_LINK_UP);
+		lgr->lgr_type = lgr_new_t;
+	} else {
+		del_llc = (struct llc_del_link_msg *)&qentry->msg;
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		qentry->msg.hd.flags |= LLC_FLAG_RESP;
+		llc_send_msg(link, &qentry->msg, LLC_SERV);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		kfree(qentry);
+		return -ENOLINK;
+	}
+	return 0;
+}
+
+int llc_cli_add_link(struct smc_link *link, struct llc_qentry *qentry,
+		     struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_link *link_new;
+	struct llc_add_link_msg *add_llc;
+	struct smc_roce_defs rocedef;
+	enum lgr_type lgr_new_t = SYMMETRIC;
+	int lnk_idx, rc;
+	int reuse_lnk_idx = SMC_MAX_SYM_LINKS + 1;
+
+	rocedef.ibdev = NULL;
+	smc_find_alt_roce_resources(&rocedef, lgr, link);
+	add_llc = (struct llc_add_link_msg *)&qentry->msg;
+	if (!memcmp(add_llc->sender_gid, link->gid_peer, 16) &&
+	    !memcmp(add_llc->sender_mac, link->mac_peer, 6)) {
+		if (!rocedef.ibdev) {
+			llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+				     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+			return 0;
+		}
+		lgr_new_t = ASYMMETRIC_PEER;
+	}
+	if (rocedef.ibdev) {
+		if (!smc_port_active(rocedef.ibdev, rocedef.port)) {
+			llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+				     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+			return 0;
+		}
+	} else {
+		lgr_new_t = ASYMMETRIC_LOCAL;
+		memcpy(&rocedef, &link->roce_defs,
+		       sizeof(struct smc_roce_defs));
+		reuse_lnk_idx = link - &lgr->lnk[0];
+	}
+	lnk_idx = llc_alloc_alt_link(lgr, lgr_new_t);
+	if (lnk_idx < 0) {
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		return 0;
+	}
+	rc = smc_create_link(lgr, lnk_idx, &rocedef);
+	if (rc) {
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		return 0;
+	}
+	link_new = &lgr->lnk[lnk_idx];
+	llc_save_add_link_info(link_new, add_llc);
+	rc = smc_ready_link(link_new);
+	if (rc) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		smc_free_link(lgr, lnk_idx);
+		return 0;
+	}
+	link_new->link_id = add_llc->link_num;
+	kfree(qentry);
+	smc_map_rmbs_to_link(lgr, lnk_idx, reuse_lnk_idx);
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	llc_send_add_link(link, &rocedef, link_new, LLC_RESP, LLC_SERV);
+	rc = llc_cli_rkey_exchange(link, link_new, smc, lnk_idx);
+	if (!rc)
+		rc = llc_cli_conf_link(link, &rocedef, link_new, lnk_idx,
+				       lgr_new_t);
+	return rc;
+}
+
+static int llc_srv_rkey_exchange(struct smc_link *link,
+				 struct smc_link *link_new,
+				 struct smc_sock *smc,
+				 int lnk_idx)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_cont_msg *addc_llc;
+	struct bufs_list_pos pos = {
+		.head = lgr->rmb_rx_bufs,
+		.ind = -1,
+		.elem = NULL,
+	};
+	int prim_lnk_idx;
+	int i;
+	u8 n, num_rkeys_send, num_rkeys_recv;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	num_rkeys_send = llc_count_rx_rmb_slot(lgr, prim_lnk_idx);
+	do {
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		llc_add_link_cont(link, lnk_idx, link_new->link_id,
+				  &num_rkeys_send, &pos);
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK_CONT);
+		if (!qentry) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return llc_get_fail_cause(lgr, LLC_SERV, smc);
+		}
+		addc_llc = (struct llc_add_link_cont_msg *)&qentry->msg;
+		num_rkeys_recv = addc_llc->num_rkeys;
+		n = addc_llc->num_rkeys;
+		for (i = 0; i < min_t(u8, n, 2); i++) {
+			llc_set_rtoken(lgr, prim_lnk_idx, lnk_idx,
+				       &addc_llc->rt[i]);
+			num_rkeys_recv--;
+		}
+		kfree(qentry);
+	} while (num_rkeys_send || num_rkeys_recv);
+	return 0;
+}
+
+static int llc_srv_conf_link(struct smc_link *link,
+			     struct smc_roce_defs *rocedef,
+			     struct smc_link *link_new, int lnk_idx,
+			     enum lgr_type lgr_new_t)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	int cas, rc;
+
+	/* send CONFIRM LINK request over the RoCE fabric */
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	rc = llc_send_confirm_link(link_new, (u8 *)&rocedef->mac,
+				   &rocedef->gid, LLC_REQ, LLC_SERV);
+	/* receive CONFIRM LINK response over the RoCE fabric */
+	if (!rc)
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_CONFIRM_LINK);
+	if (rc || !qentry) {
+		/* send DELETE LINK */
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		rc = llc_send_del_link(link, link_new, 0, LLC_DEL_LOST_PATH,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		kfree(qentry);
+		if (rc) {
+			smc_terminate_conn(lgr);
+			return rc;
+		}
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	cas = 0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+	if (!(qentry->msg.hd.flags & 0xf) ||
+	    !lgr->cache_align_size || (lgr->cache_align_size != cas)) {
+		/* no cache alignment specified, decline */
+		kfree(qentry);
+		rc = llc_send_del_link(link, link_new, 0,
+				       LLC_DEL_PROT_VIOL,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		if (rc) {
+			smc_terminate_conn(lgr);
+			return rc;
+		}
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	if (!lgr->cache_align_size)
+		lgr->cache_align_size = cas;
+	atomic_set(&link_new->state, SMC_LINK_UP);
+	lgr->lgr_type = lgr_new_t;
+	kfree(qentry);
+	return 0;
+}
+
+int llc_srv_add_link(struct smc_link *link, struct smc_roce_defs *rocedef,
+		     struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_link *link_new;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_msg *add_llc;
+	enum lgr_type lgr_new_t = SYMMETRIC;
+	int lnk_idx, rc = 0;
+	int reuse_lnk_idx = SMC_MAX_SYM_LINKS + 1;
+
+	if (rocedef->ibdev) {
+		if (!smc_port_active(rocedef->ibdev, rocedef->port))
+			return -ENOLINK;
+	} else {
+		lgr_new_t = ASYMMETRIC_LOCAL;
+		memcpy(rocedef, &link->roce_defs, sizeof(struct smc_roce_defs));
+		reuse_lnk_idx = link - &lgr->lnk[0];
+	}
+	lnk_idx = llc_alloc_alt_link(lgr, lgr_new_t);
+	if (lnk_idx < 0)
+		return 0;
+	rc = smc_create_link(lgr, lnk_idx, rocedef);
+	if (rc)
+		return rc;
+	link_new = &lgr->lnk[lnk_idx];
+	rc = llc_send_add_link(link, rocedef, link_new, LLC_REQ, LLC_SERV);
+	if (!rc) {
+		/* receive ADD LINK response over the RoCE fabric */
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK);
+	}
+	if (rc || !qentry) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		return llc_get_fail_cause(lgr, LLC_SERV, smc);
+	}
+	add_llc = (struct llc_add_link_msg *)&qentry->msg;
+	if (add_llc->hd.flags & LLC_FLAG_ADD_LNK_REJ) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		kfree(qentry);
+		return -ENOLINK;
+	}
+	if ((lgr->lgr_type == SINGLE) &&
+	    (!memcmp(add_llc->sender_gid, link->gid_peer, 16) &&
+	    !memcmp(add_llc->sender_mac, link->mac_peer, 6)))
+		lgr_new_t = ASYMMETRIC_PEER;
+	llc_save_add_link_info(link_new, add_llc);
+	kfree(qentry);
+	smc_ready_link(link_new);
+	smc_map_rmbs_to_link(lgr, lnk_idx, reuse_lnk_idx);
+	llc_srv_rkey_exchange(link, link_new, smc, lnk_idx);
+	rc = llc_srv_conf_link(link, rocedef, link_new, lnk_idx, lgr_new_t);
+	return rc;
+}
+
+static bool start_llc_agent(enum smc_role initiator,
+			    struct llc_qentry *qentry)
+{
+	struct smc_link_group *lgr = qentry->link->lnk_grp;
+
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	if (lgr->llc_ctl[initiator].active) {
+		if (((qentry->msg.hd.type == LLC_ADD_LINK) ||
+		     (qentry->msg.hd.type == LLC_DEL_LINK)) &&
+		    !lgr->delayed_q) {
+			lgr->delayed_q = qentry;
+		} else {
+			kfree(qentry);
+		}
+		spin_unlock_bh(&lgr->llc_ctl_lock);
+		return 0;
+	}
+	lgr->llc_ctl[initiator].active = llc_msg_to_grp(qentry->msg.hd.type);
+	if (qentry == lgr->delayed_q)
+		lgr->delayed_q = NULL;
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	lgr->llc_ctl[initiator].qentry = qentry;
+	return 1;
+}
+
+void llc_cancel_wait(struct smc_link *lnk)
+{
+	struct smc_link_group *lgr = lnk->lnk_grp;
+	int i;
+	bool cancelled = false;
+
+	for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+		u8 ctl_idx = lnk->pending[i].post_ctl;
+
+		if (!lnk->pending[i].conn &&
+		    (ctl_idx != LLC_NO_NOTIFY) &&
+		    lgr->llc_ctl[ctl_idx].active &&
+		    !lgr->llc_ctl[ctl_idx].qentry) {
+			lgr->llc_ctl[ctl_idx].qentry = ERR_PTR(-EPIPE);
+			cancelled = true;
+		}
+	}
+	if (cancelled)
+		wake_up_interruptible(&lgr->llc_waiter);
+}
+
+void llc_kill_waiters(struct smc_link_group *lgr)
+{
+	int i;
+	bool cancelled = false;
+
+	for (i = 0; i < LLC_NO_NOTIFY; i++) {
+		if (lgr->llc_ctl[i].active && !lgr->llc_ctl[i].qentry) {
+			lgr->llc_ctl[i].qentry = ERR_PTR(-EPIPE);
+			cancelled = true;
+		}
+	}
+	if (cancelled)
+		wake_up_interruptible(&lgr->llc_waiter);
+}
+
+static void llc_process_cli_add_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct smc_link *lnk;
+
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	mutex_lock(&lgr->conf_mutex);
+	llc_cli_add_link(lnk, qentry, NULL);
+	mutex_unlock(&lgr->conf_mutex);
+}
+
+static void llc_del_asym_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc_resp;
+	struct smc_link *link = &lgr->lnk[0];
+	int asym_lnk = SMC_MAX_SYM_LINKS + 1;
+	int i, j, k, rc;
+
+	/* determine asymmetric link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		for (j = i + 1; j <= SMC_MAX_SYM_LINKS; j++) {
+			if (!memcmp(lgr->lnk[i].roce_defs.gid.raw,
+				    lgr->lnk[j].roce_defs.gid.raw, 16)) {
+				asym_lnk = j; /* asym_lnk is i or j */
+				break;
+			}
+		}
+		if (asym_lnk < SMC_MAX_SYM_LINKS + 1)
+			break;
+	}
+	if (asym_lnk == SMC_MAX_SYM_LINKS + 1)
+		return; /* no asymmetric link */
+	for (k = 0; k <= SMC_MAX_SYM_LINKS; k++) {
+		if ((k != i) &&
+		    !memcmp(lgr->lnk[i].gid_peer, lgr->lnk[k].gid_peer, 16)) {
+			asym_lnk = i;
+			break;
+		}
+		if ((k != j) &&
+		    !memcmp(lgr->lnk[j].gid_peer, lgr->lnk[k].gid_peer, 16))
+			break;
+	}
+	if (k == SMC_MAX_SYM_LINKS + 1)
+		return; /* no asymmetric link */
+	if (asym_lnk == 0)
+		link = &lgr->lnk[1];
+	lgr->llc_ctl[LLC_SERV].active = LLC_GRP_DEL_LINK;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (!smc_lnk_downing(&lgr->lnk[asym_lnk].state))
+		return;
+	rc = smc_switch_conns(lgr, asym_lnk, 0);
+	smc_wait_no_pending_sends_on_link(&lgr->lnk[asym_lnk]);
+	if (rc == -ENOENT) {
+		smc_free_link(lgr, asym_lnk);
+		goto out;
+	}
+	rc = llc_send_del_link(link, &lgr->lnk[asym_lnk],
+			       LLC_FLAG_DEL_ORDERLY, LLC_DEL_NO_ASYM_NEEDED,
+			       LLC_SERV);
+	if (rc) {
+		if (smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+		goto out;
+	}
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO, LLC_DEL_LINK);
+	if (!qentry) {
+		if (smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+	} else {
+		del_llc_resp = (struct llc_del_link_msg *)&qentry->msg;
+		if ((ntohl(del_llc_resp->reason) != LLC_DEL_NO_ASYM_NEEDED) &&
+		    smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+		atomic_set(&lgr->lnk[asym_lnk].state, SMC_LINK_DOWN);
+	}
+	kfree(qentry);
+	smc_free_link(lgr, asym_lnk);
+out:
+	return;
+}
+
+static void llc_process_srv_add_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_add_link_fake_msg *add_llc;
+	struct smc_link *lnk;
+	struct smc_roce_defs rocdefs;
+	int rc;
+	enum lgr_type old_lgr_type;
+
+	rocdefs.ibdev = NULL;
+	rocdefs.port = 0;
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	add_llc = (struct llc_add_link_fake_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (qentry->msg.hd.flags & LLC_FLAG_LOCAL) {
+		rocdefs.ibdev = add_llc->ibdev;
+		rocdefs.port = add_llc->port_num;
+		rocdefs.vlan = lgr->vlan;
+		rc = smc_gid_by_dev(&rocdefs);
+		if (rc) {
+			kfree(qentry);
+			wake_up_interruptible(&lgr->llc_waiter);
+			return;
+		}
+	} else {
+		smc_find_alt_roce_resources(&rocdefs, lgr, lnk);
+	}
+	kfree(qentry);
+	qentry = NULL;
+	old_lgr_type = lgr->lgr_type;
+	mutex_lock(&lgr->conf_mutex);
+	rc = llc_srv_add_link(lnk, &rocdefs, NULL);
+	if (!rc && (lgr->lgr_type == SYMMETRIC)) {
+		/* delete the superfluos asymmetric link now */
+		llc_del_asym_link(lgr);
+	}
+	mutex_unlock(&lgr->conf_mutex);
+}
+
+void llc_process_add_link(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_add_link_work);
+
+	if (list_empty(&lgr->list) || (lgr->lgr_type == NONE)) {
+		/* link group to be deleted */
+		kfree(lgr->llc_ctl[LLC_SERV].qentry);
+		wake_up_interruptible(&lgr->llc_waiter);
+		goto out;
+	}
+	if (lgr->role == SMC_CLNT)
+		llc_process_cli_add_link(lgr);
+	else
+		llc_process_srv_add_link(lgr);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+out:
+	return;
+}
+
+static void llc_process_cli_del_all(struct smc_link_group *lgr,
+				    struct smc_link *link,
+				    struct llc_qentry *qentry)
+{
+	struct llc_del_link_msg *del_llc;
+	int empty;
+
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	del_llc->reason = 0;
+	mutex_lock(&lgr->conf_mutex);
+	llc_send_msg(link, &qentry->msg, LLC_SERV); /* send response */
+	memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+	mutex_unlock(&lgr->conf_mutex);
+	wake_up_interruptible(&lgr->llc_waiter);
+	/* delete link group at all */
+	spin_lock_bh(&smc_lgr_list.lock);
+	if (!list_empty(&lgr->list)) {
+		list_move_tail(&lgr->list, &smc_lgr_freeing_list);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		mutex_lock(&lgr->conf_mutex);
+		read_lock_bh(&lgr->conns_lock);
+		empty = RB_EMPTY_ROOT(&lgr->conns_all);
+		read_unlock_bh(&lgr->conns_lock);
+		mutex_unlock(&lgr->conf_mutex);
+		if (!empty)
+			smc_terminate_conn(lgr);
+		llc_kill_waiters(lgr);
+		cancel_work_sync(&lgr->llc_event_work);
+		smc_wait_no_pending_sends_on_link(link);
+		schedule_delayed_work(&lgr->free_lgr_final_work,
+				      2 * TCP_TIMEWAIT_LEN);
+	} else {
+		spin_unlock_bh(&smc_lgr_list.lock);
+	}
+}
+
+static void llc_process_cli_del_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc;
+	struct smc_link *lnk;
+	int i, link_found;
+
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	del_llc->hd.flags |= LLC_FLAG_RESP;
+	if (del_llc->hd.flags & LLC_FLAG_DEL_ALL) {
+		llc_process_cli_del_all(lgr, lnk, qentry);
+		goto out;
+	}
+	mutex_lock(&lgr->conf_mutex);
+	/* delete single link */
+	link_found = 0;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id != del_llc->link_num)
+			continue;
+		link_found = 1;
+		if (lnk->link_id == del_llc->link_num)
+			lgr->lgr_type = NONE; /* delete link of received llc */
+		else
+			llc_send_msg(lnk, &qentry->msg, LLC_SERV); /* response*/
+		if (smc_lnk_downing(&lgr->lnk[i].state)) {
+			smc_switch_conns(lgr, i, 0);
+			smc_wait_no_pending_sends_on_link(&lgr->lnk[i]);
+		}
+		if (ntohl(del_llc->reason) != LLC_DEL_NO_ASYM_NEEDED) {
+			lgr->lgr_type = SINGLE;
+			del_llc->reason = 0;
+		}
+		memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+		smc_free_link(lgr, i);
+		mutex_unlock(&lgr->conf_mutex);
+		wake_up_interruptible(&lgr->llc_waiter);
+		break;
+	}
+	if (!link_found) {
+		del_llc->reason = htonl(SMC_LLC_DEL_NOLNK);
+		llc_send_msg(lnk, &qentry->msg, LLC_SERV);
+		memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+		mutex_unlock(&lgr->conf_mutex);
+		wake_up_interruptible(&lgr->llc_waiter);
+	}
+out:
+	kfree(qentry);
+}
+
+static void llc_process_srv_del_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc;
+	struct smc_link *lnk;
+	struct smc_roce_defs rocdefs;
+	int i, lnk_idx = -1, found = 0;
+
+	mutex_lock(&lgr->conf_mutex);
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (qentry->msg.hd.flags & LLC_FLAG_DEL_ALL) {
+		/* delete entire lgr */
+		goto out;
+	}
+	/* delete single link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id == del_llc->link_num) {
+			found = 1;
+			lnk_idx = i;
+		}
+	}
+	lgr->lgr_type = SINGLE;
+	if (found && smc_lnk_downing(&lgr->lnk[lnk_idx].state)) {
+		smc_switch_conns(lgr, lnk_idx, 0);
+		smc_wait_no_pending_sends_on_link(&lgr->lnk[lnk_idx]);
+	}
+	if ((lgr->lgr_type != NONE) &&
+	    (found || (qentry->msg.hd.type == 0x80 + LLC_DEL_LINK))) {
+		struct llc_qentry *qentry2;
+
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		qentry->msg.hd.type = LLC_DEL_LINK;
+		qentry->msg.hd.flags &= ~LLC_FLAG_LOCAL;
+		llc_send_msg(lnk, &qentry->msg, LLC_SERV);
+		qentry2 = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO,
+				   LLC_DEL_LINK);
+		kfree(qentry2);
+	}
+	if (found) {
+		smc_free_link(lgr, lnk_idx);
+		if (qentry && (lgr->lgr_type == SINGLE) &&
+		    !list_empty(&lgr->list)) {
+			/* setup asymm alt link */
+			rocdefs.ibdev = NULL;
+			smc_find_alt_roce_resources(&rocdefs, lgr, lnk);
+			if (!rocdefs.ibdev) {
+				lgr->llc_ctl[LLC_SERV].active =
+							LLC_GRP_ADD_LINK;
+				llc_srv_add_link(lnk, &rocdefs, NULL);
+			}
+		}
+	}
+out:
+	mutex_unlock(&lgr->conf_mutex);
+	kfree(qentry);
+	llc_stop(lgr, LLC_GRP_DEL_LINK);
+}
+
+void llc_process_del_link(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_del_link_work);
+
+	if (lgr->role == SMC_CLNT)
+		llc_process_cli_del_link(lgr);
+	else
+		llc_process_srv_del_link(lgr);
+}
+
+void llc_process_confirm_rkey(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+					    llc_conf_rkey_work);
+	struct smc_link *lnk;
+	struct llc_qentry *qentry;
+	struct llc_confirm_rkey_msg *rk_llc;
+	int rt_idx = 0, i, rc = 0;
+	enum llc_ctl_ind initiator;
+	u8 num_entries, more = 0;
+
+	initiator = (lgr->role == SMC_SERV) ? LLC_CLNT : LLC_SERV;
+	qentry = lgr->llc_ctl[initiator].qentry;
+	lnk = qentry->link;
+	rk_llc = (struct llc_confirm_rkey_msg *)&qentry->msg;
+	lgr->llc_ctl[initiator].qentry = NULL;
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	if (lgr->llc_ctl[LLC_SERV].active > LLC_GRP_CONF_RKEY) {
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_NEG;
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_RETRY;
+	} else {
+		num_entries = rk_llc->rt[0].link_id;
+		rt_idx = smc_get_rtoken(lgr);
+		llc_set_rtoken_by_link(lgr, lnk->link_id, rt_idx,
+				       &rk_llc->rt[0]);
+		for (i = 1; i <= min_t(u8, num_entries, 2); i++)
+			llc_set_rtoken_by_link(lgr, rk_llc->rt[i].link_id,
+					       rt_idx, &rk_llc->rt[i]);
+		more = (num_entries > 2) ? 1 : 0;
+	}
+	rc = llc_send_msg(lnk, &qentry->msg, LLC_NO_NOTIFY);
+	kfree(qentry);
+again:
+	if (more && (rc >= 0)) {
+		qentry = llc_wait(lgr, (enum smc_role)initiator, LLC_WAIT_TIMEO,
+				  LLC_CONFIRM_RKEY_CONT);
+		if (!qentry) {
+			more = 0;
+		} else {
+			struct llc_conf_rkey_cont_msg *rkc_llc =
+				(struct llc_conf_rkey_cont_msg *)&qentry->msg;
+			lgr->llc_ctl[initiator].qentry = NULL;
+			qentry->msg.hd.flags |= LLC_FLAG_RESP;
+			num_entries = rkc_llc->num_rkeys;
+			for (i = 0; i <= min_t(u8, num_entries, 2); i++) {
+				llc_set_rtoken_by_link(lgr,
+						       rkc_llc->rt[i].link_id,
+						       rt_idx, &rkc_llc->rt[i]);
+			}
+			more = (num_entries > 3) ? 1 : 0;
+			rc = llc_send_msg(lnk, &qentry->msg, LLC_NO_NOTIFY);
+			kfree(qentry);
+		}
+		goto again;
+	}
+	memset(&lgr->llc_ctl[initiator], 0, sizeof(struct llc_ctl));
+}
+
+void llc_process_delete_rkey(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_del_rkey_work);
+	struct smc_link *lnk;
+	struct llc_qentry *qentry;
+	struct llc_delete_rkey_msg *drk_llc;
+	int initiator;
+
+	initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+	qentry = lgr->llc_ctl[initiator].qentry;
+	lnk = qentry->link;
+	drk_llc = (struct llc_delete_rkey_msg *)&qentry->msg;
+	lgr->llc_ctl[initiator].qentry = NULL;
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	if (lgr->llc_ctl[LLC_SERV].active > LLC_GRP_CONF_RKEY) {
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_NEG;
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_RETRY;
+	} else {
+		int i, rc;
+		u8 local_mask = 0, err_val;
+
+		err_val = 0x80;
+		for (i = 0; i < min_t(u8, drk_llc->num_rkeys, 8); i++) {
+			rc = smc_clear_rtokens_by_link(lgr, lnk->link_id,
+						       drk_llc->rk[i]);
+			if (rc < 0)
+				local_mask = local_mask + err_val;
+			err_val = err_val / 2;
+		}
+		if (local_mask) {
+			drk_llc->hd.flags |= LLC_FLAG_RKEY_NEG;
+			drk_llc->err_mask = local_mask;
+		}
+	}
+	llc_send_msg(lnk, &qentry->msg, initiator);
+	kfree(qentry);
+	memset(&lgr->llc_ctl[initiator], 0, sizeof(struct llc_ctl));
+}
+
+static void llc_handle_event(struct llc_qentry *qentry,
+			     struct smc_link_group *lgr)
+{
+	struct smc_link *lnk = qentry->link;
+	struct llc_ctl *srv_ctl = &lgr->llc_ctl[LLC_SERV];
+	int initiator;
+
+	switch (qentry->msg.hd.type) {
+	case LLC_ADD_LINK:
+		if (list_empty(&lgr->list)) /* link group to be freed */
+			break;
+		if (lgr->role == SMC_CLNT) {
+			if (srv_ctl->active == LLC_GRP_ADD_LINK) {
+				/* some process is waiting for this response */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_add_link_work);
+			}
+		} else if (start_llc_agent(SMC_SERV, qentry)) {
+			/* else on SERV handle client suggest */
+			schedule_work(&lgr->llc_add_link_work);
+		}
+		break;
+	case LLC_DEL_LINK:
+		if (lgr->role == SMC_CLNT) {
+			if (srv_ctl->active) {
+				/* DEL LINK REQ during ADD LINK SEQ */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_del_link_work);
+			}
+		} else {
+			if (qentry->msg.hd.flags & LLC_FLAG_DEL_ALL) {
+				smc_terminate_conn(lgr);
+				kfree(qentry);
+			} else if ((srv_ctl->active == LLC_GRP_ADD_LINK) &&
+				   (!srv_ctl->qentry)) {
+				/* DEL LINK REQ during ADD LINK SEQ */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_del_link_work);
+			}
+		}
+		break;
+	case LLC_CONFIRM_RKEY:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		if (start_llc_agent(initiator, qentry))
+			schedule_work(&lgr->llc_conf_rkey_work);
+		break;
+	case LLC_DELETE_RKEY:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		if (start_llc_agent(initiator, qentry))
+			schedule_work(&lgr->llc_del_rkey_work);
+		break;
+	case LLC_CONFIRM_RKEY_CONT:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		lgr->llc_ctl[initiator].qentry = qentry;
+		wake_up_interruptible(&lgr->llc_waiter);
+		break;
+	case LLC_CONFIRM_LINK:
+	case LLC_ADD_LINK_CONT:
+		if (lgr->llc_ctl[LLC_SERV].active) {
+			/* some process is waiting for this response */
+			srv_ctl->qentry = qentry;
+			wake_up_interruptible(&lgr->llc_waiter);
+		}
+		break;
+	case LLC_TEST_LINK:
+		qentry->msg.hd.flags |= LLC_FLAG_RESP;
+		llc_send_msg(qentry->link, &qentry->msg, LLC_NO_NOTIFY);
+		kfree(qentry);
+		break;
+	case LLC_NWM_DATA:
+		lgr->nwm_flags = qentry->msg.hd.flags;
+		if (qentry->msg.hd.flags & LLC_FLAG_HOSTNAME)
+			strncpy(lgr->nwm_data, qentry->msg.data,
+				sizeof(lgr->nwm_data));
+		else
+			memcpy(lgr->nwm_data, qentry->msg.data,
+			       sizeof(lgr->nwm_data));
+		kfree(qentry);
+		break;
+	default:
+		if (!(qentry->msg.hd.type & LLC_OPT_MSG_CTRL)) {
+			/* LLC protocol violation */
+			llc_send_del_link(lnk, NULL, 0,
+					  LLC_DEL_PROT_VIOL, LLC_NO_NOTIFY);
+			smc_terminate_conn(lgr);
+		}
+		kfree(qentry);
+		break;
+	}
+}
+
+void llc_event_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_event_work);
+	struct llc_qentry *qentry;
+
+	if ((!lgr->llc_ctl[LLC_SERV].active) &&
+	    (lgr->delayed_q) &&
+		(atomic_read(&lgr->delayed_q->link->state) != SMC_LINK_FREED))
+		llc_handle_event(lgr->delayed_q, lgr);
+
+again:
+	spin_lock_bh(&lgr->llc_event_q_lock);
+	if (!list_empty(&lgr->llc_event_q)) {
+		qentry = list_first_entry(&lgr->llc_event_q, struct llc_qentry,
+					  list);
+		list_del_init(&qentry->list);
+		spin_unlock_bh(&lgr->llc_event_q_lock);
+		llc_handle_event(qentry, lgr);
+		goto again;
+	}
+	spin_unlock_bh(&lgr->llc_event_q_lock);
+}
+
diff --git a/net/smc/smc_llc.h b/net/smc/smc_llc.h
new file mode 100644
index 0000000..f47c499
--- /dev/null
+++ b/net/smc/smc_llc.h
@@ -0,0 +1,192 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Definitions for LLC message handling
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Klaus Wacker <Klaus.Wacker@de.ibm.com>
+ *              Ursula Braun <Ursula Braun@de.ibm.com>
+ */
+
+#define LLC_DATA_LEN    40
+#define LLC_SEND_POSTED		0xffffffff
+
+#define LLC_FLAG_RESP		0x80
+#define LLC_FLAG_HOSTNAME	0x80
+#define LLC_FLAG_RKEY_DEL	0x40
+#define LLC_FLAG_RKEY_NEG	0x20
+#define LLC_FLAG_RKEY_RETRY	0x10
+#define LLC_FLAG_DEL_ORDERLY	0x20
+#define LLC_FLAG_DEL_ALL	0x40
+#define LLC_FLAG_ADD_LNK_REJ	0x40
+#define LLC_FLAG_NO_ALT_PATH	0x01
+#define LLC_FLAG_LOCAL		0x04    /* Linux-only enhancement */
+#define LLC_FLAGS_DEL_ALL_ORD	0x60
+
+/* LLC DELETE LINK Request Reason Codes */
+#define LLC_DEL_LOST_PATH	0x00010000
+#define LLC_DEL_OP_INIT_TERM	0x00020000
+#define LLC_DEL_PROG_INIT_TERM	0x00030000
+#define LLC_DEL_PROT_VIOL	0x00040000
+#define LLC_DEL_NO_ASYM_NEEDED	0x00050000
+
+enum llc_reqresp {
+	LLC_REQ,
+	LLC_RESP
+};
+
+enum llc_msg_type {
+	LLC_CONFIRM_LINK		= 0x01,
+	LLC_ADD_LINK			= 0x02,
+	LLC_ADD_LINK_CONT		= 0x03,
+	LLC_DEL_LINK			= 0x04,
+	LLC_CONFIRM_RKEY		= 0x06,
+	LLC_CONFIRM_RKEY_CONT		= 0x08,
+	LLC_DELETE_RKEY			= 0x09,
+	LLC_TEST_LINK			= 0x07,
+	LLC_OPT_MSG_CTRL		= 0x80,
+	LLC_NWM_DATA			= 0x8A,
+	RMBE_CTRL			= 0xFE,
+};
+
+struct rmb_rtoken {		/* RMB credentials given to peer */
+	__be32 rmb_key;		/* rkey */
+	__be64 rmb_vaddr;	/* RDMA virtual address */
+} __packed;
+
+struct rmb_rtoken2 {
+	__be32 rmb_key;
+	__be32 rmb_key_new;
+	__be64 rmb_vaddr_new;
+} __packed;
+
+struct rmb_rtoken3 {
+	u8   link_id;
+	__be32 rmb_key;
+	__be64 rmb_vaddr;
+} __packed;
+
+struct llc_hdr {
+	u8 type;
+	u8 length;	/* 44 */
+	u8 version;	/* 0x01 */
+	u8 flags;
+} __packed;
+
+struct llc_msg {
+	struct llc_hdr hd;
+	u8 data[LLC_DATA_LEN];
+} __packed;
+
+struct llc_confirm_msg {		/* type 0x01 */
+	struct llc_hdr hd;
+	u8 sender_mac[6];
+	u8 sender_gid[16];
+	u8 sender_qp_num[3];
+	u8 link_num;
+	__be32 link_uid;
+	u8 max_links;
+	u8 reserved[9];
+} __packed;
+
+struct llc_add_link_msg {		/* type 0x02 */
+	struct llc_hdr hd;
+	u8 sender_mac[6];
+	u8 reserved2[2];
+	u8 sender_gid[16];
+	u8 sender_qp_num[3];
+	u8 link_num;
+	u8 flags2;	/* QP mtu */
+	u8 initial_psn[3];
+	u8 reserved[8];
+} __packed;
+
+struct llc_add_link_cont_msg {	/* type 0x03 */
+	struct llc_hdr hd;
+	u8 link_num;
+	u8 num_rkeys;
+	u8 reserved2[2];
+	struct rmb_rtoken2 rt[2];
+	u8 reserved[4];
+} __packed;
+
+struct llc_del_link_msg {		/* type 0x04 */
+	struct llc_hdr hd;
+	u8 link_num;
+	__be32 reason;
+	u8 reserved[35];
+} __packed;
+
+struct llc_confirm_rkey_msg {	/* type 0x05 */
+	struct llc_hdr hd;
+	struct rmb_rtoken3 rt[3];
+	u8 reserved[1];
+} __packed;
+
+struct llc_conf_rkey_cont_msg {	/* type 0x08 */
+	struct llc_hdr hd;
+	u8 num_rkeys;
+	struct rmb_rtoken3 rt[3];
+} __packed;
+
+struct llc_delete_rkey_msg {	/* type 0x09 */
+	struct llc_hdr hd;
+	u8 num_rkeys;
+	u8 err_mask;
+	u8 reserved[2];
+	__be32 rk[8];
+	u8 reserved2[4];
+} __packed;
+
+struct llc_test_link_msg {		/* type 0x07 */
+	struct llc_hdr hd;
+	u8 user_data[16];
+	u8 reserved[24];
+} __packed;
+
+struct llc_nwm_data_msg {		/* type 0x8A */
+	struct llc_hdr hd;
+	u8 ident[16];
+	u8 reserved[24];
+} __packed;
+
+struct llc_add_link_fake_msg {		/* type 0x82 */
+	struct llc_hdr hd;
+	struct smc_ib_device *ibdev;
+	u8 port_num;
+	u8 reserved[31];
+} __packed;
+
+struct llc_qentry {
+	struct list_head list;
+	struct smc_link *link;
+	struct llc_msg msg;
+};
+
+void llc_init_workers(struct smc_link_group *);
+int llc_enqueue(struct smc_link *, struct llc_msg *);
+int llc_initiate(struct smc_link_group *, enum llc_msg_group);
+void llc_stop(struct smc_link_group *, int);
+struct llc_qentry *llc_wait(struct smc_link_group *, enum smc_role, int, u8);
+int llc_send_confirm_link(struct smc_link *, u8 *, union ib_gid *,
+			  enum llc_reqresp, enum llc_ctl_ind);
+int llc_do_confirm_rkey(struct smc_link *, struct rmb_rx_addrs *);
+int llc_send_test_link(struct smc_link *, u8 *);
+int llc_send_add_link(struct smc_link *, struct smc_roce_defs *,
+		      struct smc_link *, enum llc_reqresp, enum llc_ctl_ind);
+int llc_send_del_link(struct smc_link *, struct smc_link *, u8, u32,
+		      enum llc_ctl_ind);
+int llc_do_del_rkey(struct smc_link *, struct rmb_rx_addrs *);
+void llc_event_worker(struct work_struct *);
+void llc_process_add_link(struct work_struct *);
+void llc_process_del_link(struct work_struct *);
+void llc_process_confirm_rkey(struct work_struct *);
+void llc_process_delete_rkey(struct work_struct *);
+int llc_get_fail_cause(struct smc_link_group *, enum llc_ctl_ind,
+		       struct smc_sock *);
+int llc_cli_add_link(struct smc_link *, struct llc_qentry *, struct smc_sock *);
+int llc_srv_add_link(struct smc_link *, struct smc_roce_defs *,
+		     struct smc_sock *);
+void llc_kill_waiters(struct smc_link_group *);
+void llc_cancel_wait(struct smc_link *);
diff --git a/net/smc/smc_proc.c b/net/smc/smc_proc.c
new file mode 100644
index 0000000..70b94c2
--- /dev/null
+++ b/net/smc/smc_proc.c
@@ -0,0 +1,953 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  auxiliary functions
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              based on prototype from Frank Blaschka
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/seq_file.h>
+#include <linux/debugfs.h>
+#include <linux/uaccess.h>
+#include <linux/workqueue.h>
+#include <linux/proc_fs.h>
+#include <linux/netdevice.h>
+#include <linux/if_vlan.h>
+#include <linux/inetdevice.h>
+#include <linux/types.h>
+#include <net/sock.h>
+#include <net/inet_sock.h>
+#include <net/tcp.h>
+#include <net/route.h>
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <rdma/ib_addr.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+struct smc_pnet_entry {
+	struct list_head list;
+	char dev_name[IFNAMSIZ];
+	char net_name[SMC_MAX_PNET_ID_LEN];
+	u8 port;
+	enum {
+		PNET_ETH,
+		PNET_IB
+	} type;
+	__be32 ipnet;			/* IPv4 address network part */
+	__be32 ipaddr;			/* IPv4 address */
+};
+
+static struct proc_dir_entry *procfs_root;
+static struct proc_dir_entry *procfs_pnet_conf;
+static struct proc_dir_entry *procfs_buff_cnt;
+static struct proc_dir_entry *procfs_max_conn_per_lgr;
+static struct proc_dir_entry *procfs_port_add_delay;
+static struct proc_dir_entry *procfs_sndbuf;
+static struct proc_dir_entry *procfs_rcvbuf;
+static struct list_head smc_pnet_list;
+static rwlock_t smc_pnet_list_lock;
+
+/* determine first gid of roce device  and derive mac from it */
+int smc_get_ib_mac(struct smc_roce_defs *rocdefs)
+{
+	int rc = 0;
+
+	rc = ib_query_gid(rocdefs->ibdev->dev, rocdefs->port, 0, &rocdefs->gid);
+	if (!rc) {
+		memcpy(&rocdefs->mac[0], &rocdefs->gid.raw[8], 3);
+		memcpy(&rocdefs->mac[3], &rocdefs->gid.raw[13], 3);
+		rocdefs->mac[0] &= ~0x02;
+	}
+	return rc;
+}
+
+int smc_port_active(struct smc_ib_device *dev, u8 port)
+{
+	struct ib_port_attr props;
+	int rc;
+
+	rc = ib_query_port(dev->dev, port, &props);
+	if (rc)
+		return 0;
+	if (props.state != IB_PORT_ACTIVE)
+		return 0;
+	return 1;
+}
+
+static void smc_ib_dev_by_pnet(struct smc_roce_defs *rocdefs, char *pnet,
+			       struct smc_link *link)
+{
+	struct smc_ib_device *dev = NULL;
+	struct smc_pnet_entry *entry;
+	int found = 0;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if ((entry->type != PNET_IB) ||
+		    strncmp(pnet, entry->net_name, sizeof(entry->net_name)))
+			continue;
+		if (link &&
+		    !strncmp(entry->dev_name,
+			     link->roce_defs.ibdev->dev->name,
+			     sizeof(entry->dev_name)) &&
+		    (entry->port == link->roce_defs.port)) {
+			continue;
+		}
+		list_for_each_entry(dev, &smc_ib_devices, list) {
+			if (!strncmp(dev->dev->name, entry->dev_name,
+				     sizeof(entry->dev_name))) {
+				read_unlock(&smc_pnet_list_lock);
+				found = smc_port_active(dev, entry->port);
+				read_lock(&smc_pnet_list_lock);
+				if (found)
+					break;
+			}
+		}
+		if (found)
+			break;
+	}
+	read_unlock(&smc_pnet_list_lock);
+
+	if (!found)
+		return;
+
+	rocdefs->ibdev = dev;
+	rocdefs->port = entry->port;
+}
+
+int smc_pnet_by_ibdev(u8 port, char *pnet, struct smc_ib_device *dev)
+{
+	struct smc_pnet_entry *entry;
+	int rc = -EINVAL;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_IB &&
+		    !strncmp(dev->dev->name, entry->dev_name,
+			     sizeof(entry->dev_name)) &&
+		    (port == entry->port)) {
+			rc = 0;
+			strncpy(pnet, entry->net_name, sizeof(entry->net_name));
+			break;
+		}
+	}
+	read_unlock(&smc_pnet_list_lock);
+	return rc;
+}
+
+/* Scan the PNET list and find all entries with different PNET name but
+ * same IP address network part.
+ * Return -EINVAL when identical network part and different PNET ids.
+ */
+static int smc_pnet_checkipnet(struct smc_pnet_entry *hit)
+{
+	int rc = 0;
+	struct smc_pnet_entry *entry;
+
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type != PNET_ETH || hit == entry)
+			continue;
+		if (hit->ipnet == entry->ipnet &&
+		    strcmp(hit->net_name, entry->net_name))
+			rc = -EINVAL;
+	}
+	return rc;
+}
+
+/* Get first primary address of interface listed in PNET table.
+ * Returns number of primary interface addresses detected.
+ */
+static int smc_pnet_get1st(struct smc_pnet_entry *entry, struct net_device *dev)
+{
+	int cnt = 0;
+
+	for_primary_ifa(dev->ip_ptr) {
+		if (!entry->ipaddr) {
+			entry->ipnet = ifa->ifa_address & ifa->ifa_mask;
+			entry->ipaddr = ifa->ifa_address;
+		}
+		++cnt;
+	} endfor_ifa(dev->ip_ptr);
+	return cnt;
+}
+
+/* Get net_device assigned to an interface and find the first IP address and
+ * IP network address.
+ *
+ * Note:
+ * Use the same network namespace as the incoming (destination) device. This
+ * should be identical to the process network namespace.
+ */
+static void smc_pnet_getipdata(struct smc_pnet_entry *entry, struct net *ns)
+{
+	struct net_device *dev = dev_get_by_name(ns, entry->dev_name);
+
+	entry->ipnet = 0;
+	entry->ipaddr = 0;
+	if (dev) {
+		smc_pnet_get1st(entry, dev);
+		dev_put(dev);
+	}
+}
+
+static int smc_pnet_by_tcpsk(char *pnet, struct sock *sk)
+{
+	int found = -EINVAL;
+	struct smc_pnet_entry *entry, *hit = NULL;
+	struct dst_entry *dst = sk_dst_get(sk);
+
+	if (!dst)
+		goto out;
+	if (!dst->dev)
+		goto out_rel;
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_ETH) {
+			smc_pnet_getipdata(entry, dev_net(dst->dev));
+			if (found == -EINVAL &&
+			    !strncmp(dst->dev->name, entry->dev_name,
+				     sizeof(entry->dev_name))) {
+				strncpy(pnet, entry->net_name,
+					sizeof(entry->net_name));
+				found = 0;
+				hit = entry;
+			}
+		}
+	}
+	if (hit)
+		found = smc_pnet_checkipnet(hit);
+	read_unlock(&smc_pnet_list_lock);
+out_rel:
+	dst_release(dst);
+out:
+	return found;
+}
+
+int smc_gid_by_dev(struct smc_roce_defs *rocdefs)
+{
+	int rc = 0;
+
+	rc = ib_query_gid(rocdefs->ibdev->dev, rocdefs->port, 0,
+			  &rocdefs->gid);
+	rocdefs->sgid_idx = 0;
+	smc_get_ib_mac(rocdefs);
+	return rc;
+}
+
+static int smc_gid_by_tcpsk(struct smc_roce_defs *rocdefs, struct sock *sk)
+{
+	struct dst_entry *dst = sk_dst_get(sk);
+	int rc;
+
+	rocdefs->vlan = 0;
+	if (dst) {
+		if (dst->dev) {
+			if (is_vlan_dev(dst->dev))
+				rocdefs->vlan = vlan_dev_vlan_id(dst->dev);
+		}
+		dst_release(dst);
+	}
+	rc = smc_gid_by_dev(rocdefs);
+	return rc;
+}
+
+void smc_find_roce_resources(struct smc_roce_defs *rocdefs, char *pnet,
+			     struct sockaddr_in *inaddr, struct sock *sk)
+{
+	int rc;
+
+	rocdefs->ibdev = NULL;
+	rocdefs->port = 0;
+	rc = smc_pnet_by_tcpsk(pnet, sk);
+	if (rc) {
+		/* add fallback path and try to find ib dev/port
+		 * via mac address of tcp sock
+		 */
+		return;
+	}
+	smc_ib_dev_by_pnet(rocdefs, pnet, NULL);
+	if (!rocdefs->ibdev)
+		return;
+
+	rc = smc_gid_by_tcpsk(rocdefs, sk);
+	if (rc) {
+		rocdefs->ibdev = NULL;
+		return;
+	}
+}
+
+void smc_find_alt_roce_resources(struct smc_roce_defs *rocdefs,
+				 struct smc_link_group *lgr,
+				 struct smc_link *link)
+{
+	int rc;
+
+	smc_ib_dev_by_pnet(rocdefs, lgr->pnet_id, link);
+	if (!rocdefs->ibdev)
+		return;
+
+	rocdefs->vlan = lgr->vlan;
+	rc = smc_gid_by_dev(rocdefs);
+	if (rc) {
+			rocdefs->ibdev = NULL;
+			return;
+	}
+}
+
+int smc_netinfo_by_tcpsk(__be32 *subnet, u8 *mlen, struct socket *tcpsocket)
+{
+	struct dst_entry *dst = sk_dst_get(tcpsocket->sk);
+	struct sockaddr_in addr;
+	int rc = -ENODEV;
+	int len;
+
+	if (!dst)
+		goto out;
+	if (!dst->dev)
+		goto out_rel;
+	kernel_getsockname(tcpsocket, (struct sockaddr *)&addr, &len);
+	for_ifa(dst->dev->ip_ptr) {
+		if (ifa->ifa_address != addr.sin_addr.s_addr)
+			continue;
+		*mlen = inet_mask_len(ifa->ifa_mask);
+		*subnet = ifa->ifa_address & ifa->ifa_mask;
+		rc = 0;
+		break;
+	} endfor_ifa(dst->dev->ip_ptr);
+out_rel:
+	dst_release(dst);
+out:
+	return rc;
+}
+
+static void smc_print_area(struct seq_file *m, char *buf, int max_len,
+			   int pr_len)
+{
+	char fmtbuf[60];
+	int i, len;
+
+	len = min_t(size_t, max_len, pr_len);
+	while (len > 0) {
+		i = (len < 16) ? len : 16;
+		hex_dump_to_buffer(buf, i, 16, 4, fmtbuf, 60, 0);
+		seq_printf(m, "%p: %s\n", buf, fmtbuf);
+		buf += 16;
+		len -= 16;
+	}
+}
+
+static int smc_sock_show(struct seq_file *m, void *v)
+{
+	struct smc_proc_dir_entry *proc = m->private;
+	struct sock *sk;
+	struct smc_sock *smc;
+	struct sockaddr_in locl_addr, peer_addr;
+	int rc, len;
+
+	if (!proc)
+		return 0;
+
+	sk = proc->sk;
+	if (!sk)
+		return 0;
+
+	sock_hold(sk);
+	if (sock_flag(sk, SOCK_DEAD) || (sk->sk_state == SMC_DESTRUCT))
+		goto out;
+
+	smc = smc_sk(sk);
+	seq_printf(m, "state: %d\n", sk->sk_state);
+	seq_printf(m, "uid: %d\n",
+		   from_kuid_munged(seq_user_ns(m), sock_i_uid(sk)));
+	seq_printf(m, "inode: %ld\n", sock_i_ino(sk));
+
+	if (sk->sk_state == SMC_INIT)
+		goto out;
+
+	if ((smc->tcpsocket) && (smc->tcpsocket->sk)) {
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket,
+						  (struct sockaddr *)&locl_addr,
+						  &len, 0);
+		if (!rc) {
+			seq_printf(m, "local_addr: %pI4\n",
+				   &locl_addr.sin_addr.s_addr);
+			seq_printf(m, "local_port: %d\n",
+				   ntohs(locl_addr.sin_port));
+		}
+	}
+
+	if (sk->sk_state == SMC_LISTEN)
+		goto out;
+
+	if ((smc->tcpsocket) && (smc->tcpsocket->sk)) {
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket,
+						  (struct sockaddr *)&peer_addr,
+						  &len, 1);
+		if (!rc) {
+			seq_printf(m, "peer_addr: %pI4\n",
+				   &peer_addr.sin_addr.s_addr);
+			seq_printf(m, "peer_port: %d\n",
+				   ntohs(peer_addr.sin_port));
+		}
+	}
+
+	seq_printf(m, "tcp_fallback: %d\n", smc->use_tcp);
+	if (smc->use_tcp)
+		goto out;
+
+	if (smc->conn.lgr && (sk->sk_state != SMC_CLOSED)) {
+		seq_printf(m, "link group target: %#x\n",
+			   smc->conn.lgr->daddr);
+		seq_printf(m, "link group role: %d\n", smc->conn.lgr->role);
+		seq_puts(m, "gid_peer:\n");
+		smc_print_area(m, smc->conn.lgr->lnk[0].gid_peer, 16, 16);
+		smc_print_area(m, smc->conn.lgr->lnk[1].gid_peer, 16, 16);
+	}
+	seq_printf(m, "tx buffer len: %d\n", smc->conn.rmb_tx_size);
+	seq_printf(m, "rx buffer len: %d\n", smc->conn.rmb_rx_size);
+	seq_printf(m, "local alert token: %#x\n", smc->conn.alert_token_local);
+
+	seq_printf(m, "peer rx buffer len: %d\n", smc->conn.peer_rx_buf_len);
+	seq_printf(m, "local rx pc %d pw %d cc %d cw %d\n",
+		   smc->conn.local_rx_ctrl.p_curs.s.curs.c,
+		   smc->conn.local_rx_ctrl.p_curs.s.curs.w,
+		   smc->conn.local_rx_ctrl.c_curs.s.curs.c,
+		   smc->conn.local_rx_ctrl.c_curs.s.curs.w);
+	seq_printf(m, "local tx pc %d pw %d cc %d cw %d\n",
+		   smc->conn.local_tx_ctrl.p_curs.s.curs.c,
+		   smc->conn.local_tx_ctrl.p_curs.s.curs.w,
+		   smc->conn.local_tx_ctrl.c_curs.s.curs.c,
+		   smc->conn.local_tx_ctrl.c_curs.s.curs.w);
+	seq_printf(m, "tx producer_flags %#x conn_state_flags %#x\n",
+		   *(u8 *)&smc->conn.local_tx_ctrl.p_flags,
+		   *(u8 *)&smc->conn.local_tx_ctrl.conn_state_flags);
+	seq_printf(m, "rx producer_flags %#x conn_state_flags %#x\n",
+		   *(u8 *)&smc->conn.local_rx_ctrl.p_flags,
+		   *(u8 *)&smc->conn.local_rx_ctrl.conn_state_flags);
+
+	if (smc->conn.rmb_tx_elem &&
+	    smc->conn.rmb_tx_elem->rmb_tx &&
+	    smc->conn.rmb_tx_elem->rmb_tx) {
+		seq_puts(m, "tx buffer:\n");
+		smc_print_area(m, smc->conn.rmb_tx_elem->rmb_tx/*->buffer*/,
+			       smc->conn.rmb_tx_size, 15000);
+	}
+	if (smc->conn.rmb_rx_elem &&
+	    smc->conn.rmb_rx_elem->rmb_rx &&
+	    smc->conn.rmb_rx_elem->rmb_rx) {
+		seq_puts(m, "rx buffer:\n");
+		smc_print_area(m, smc->conn.rmb_rx_elem->rmb_rx/*->buffer*/,
+			       smc->conn.rmb_rx_size, 15000);
+	}
+
+out:
+	sock_put(sk);
+	return 0;
+}
+
+static int smc_sock_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_sock_show, PDE_DATA(inode));
+}
+
+static const struct file_operations procfs_smc_sock_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_sock_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.release = single_release,
+};
+
+int smc_sock_proc_create(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+
+	sprintf(smc->proc_name, "Sx%p_0x%p", sk, smc->proc);
+	smc->proc->proc = proc_create_data(smc->proc_name,
+					   S_IFREG | S_IRUGO | S_IWUSR,
+					   procfs_root, &procfs_smc_sock_fops,
+					   smc->proc);
+	return 0;
+}
+
+static int smc_pnet_show(struct seq_file *m, void *v)
+{
+	struct smc_pnet_entry *entry;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_IB)
+			seq_printf(m, "%-*s ib  %s %d\n", SMC_MAX_PNET_ID_LEN,
+				   entry->net_name, entry->dev_name,
+				   entry->port);
+		else
+			seq_printf(m, "%-*s eth %s\n", SMC_MAX_PNET_ID_LEN,
+				   entry->net_name, entry->dev_name);
+	}
+	read_unlock(&smc_pnet_list_lock);
+	return 0;
+}
+
+static struct smc_pnet_entry *smc_pnet_create_entry(int argc, char *argv[])
+{
+	struct smc_pnet_entry *entry;
+	long rc = 0;
+
+	entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+	if (!entry) {
+		rc = -ENOMEM;
+		goto out;
+	}
+	rc = -EINVAL;
+	if (!strcmp(argv[2], "ib")) {
+		if ((argc != 5) || kstrtou8(argv[4], 0, &entry->port))
+			goto out;
+		entry->type = PNET_IB;
+	} else if (!strcmp(argv[2], "eth")) {
+		if (argc > 4)
+			goto out;
+		entry->type = PNET_ETH;
+	} else {
+		goto out;
+	}
+	if (strlen(argv[1]) >= sizeof(entry->net_name) ||
+	    strlen(argv[3]) >= sizeof(entry->dev_name))
+		goto out;
+	strncpy(entry->net_name, argv[1], sizeof(entry->net_name));
+	strncpy(entry->dev_name, argv[3], sizeof(entry->dev_name));
+	return entry;
+
+out:
+	kfree(entry);
+	return ERR_PTR(rc);
+}
+
+/* Update the PNET ID table. Make sure entries are sorted by PNET ID
+ * for fancy output.
+ * @entry: entry to search for
+ * @del: delete existing entry if found, otherwise add.
+ *
+ * Returns 0 on success, <0 otherwise.
+ */
+static int smc_pnet_update(struct smc_pnet_entry *entry, bool del)
+{
+	struct smc_pnet_entry *tmp_entry, *n_entry;
+	int rc = -EINVAL;
+	int found = 0;
+
+	/* keep list sorted by PNET ID */
+	write_lock(&smc_pnet_list_lock);
+	list_for_each_entry_safe(tmp_entry, n_entry, &smc_pnet_list, list) {
+		if (entry->type == tmp_entry->type &&
+		    !strncmp(entry->dev_name, tmp_entry->dev_name,
+		    sizeof(entry->dev_name)) &&
+		    entry->port == tmp_entry->port) {
+			/* entry for this device, port and type found */
+			if (del) {
+				list_del(&tmp_entry->list);
+				kfree(tmp_entry);
+				kfree(entry);
+				rc = 0;
+			}
+			found = 1;
+			break;
+		}
+		if (strncmp(entry->net_name, tmp_entry->net_name,
+			    sizeof(entry->net_name)) < 0)
+			break;
+	}
+	if (!del && !found) {
+		rc = 0;
+		if (entry->type == PNET_ETH) {
+			smc_pnet_getipdata(entry, current->nsproxy->net_ns);
+			rc = smc_pnet_checkipnet(entry);
+		}
+		if (!rc)
+			list_add_tail(&entry->list, &tmp_entry->list);
+	}
+	write_unlock(&smc_pnet_list_lock);
+
+	return rc;
+}
+
+/* add|del <PNET ID> eth|ib <device name> [<port>] */
+static ssize_t smc_pnet_write(struct file *file,
+			      const char __user *user_buffer,
+			      size_t count, loff_t *offset)
+{
+	char buf[128];
+	char *pos;
+	char *tok;
+	char *tmp;
+	char *argv[5];
+	int argc, rc = -EINVAL, del;
+	size_t len = min_t(size_t, count, sizeof(buf) - 1);
+	struct smc_pnet_entry *entry;
+
+	memset(buf, 0, sizeof(buf));
+	argc = 0;
+
+	if (copy_from_user(buf, user_buffer, len)) {
+		rc = -EFAULT;
+		goto out;
+	}
+	tmp = buf;
+	pos = strsep(&tmp, "\n");
+
+	while ((tok = strsep(&pos, " "))) {
+		if (!strcmp(tok, ""))
+			continue;
+		if (argc == 5)
+			break;
+		argv[argc] = tok;
+		argc++;
+	}
+
+	if (argc < 4 || tok)
+		goto out;
+
+	if (!strcmp(argv[0], "add"))
+		del = 0;
+	else if (!strcmp(argv[0], "del"))
+		del = 1;
+	else
+		goto out;
+	entry = smc_pnet_create_entry(argc, argv);
+	if (IS_ERR(entry)) {
+		rc = PTR_ERR(entry);
+		goto out;
+	}
+	rc = smc_pnet_update(entry, del);
+	if (!rc)
+		rc = count;
+out:
+	return rc;
+}
+
+static int smc_pnet_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_pnet_show, PDE_DATA(inode));
+}
+
+static int smc_buff_cnt_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%u\n", smc_ctrl_buf_cnt);
+	return 0;
+}
+
+static ssize_t smc_buff_cnt_write(struct file *file,
+				  const char __user *user_buffer,
+				  size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 3) ||
+	    (tmp > SMC_MAX_WRE) ||
+	    ((smc_max_conn_per_lgr * 10 * tmp) > SMC_MAX_CQE)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	spin_lock_bh(&smc_lgr_list.lock);
+	if (!list_empty(&smc_lgr_list.list)) {
+		rc = -EPERM;
+		goto out_unlock;
+	}
+	smc_ctrl_buf_cnt = tmp;
+	rc = count;
+out_unlock:
+	spin_unlock_bh(&smc_lgr_list.lock);
+out:
+	return rc;
+}
+
+static int smc_buff_cnt_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_buff_cnt_show, PDE_DATA(inode));
+}
+
+static int smc_max_conn_per_lgr_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_max_conn_per_lgr);
+	return 0;
+}
+
+static ssize_t smc_max_conn_per_lgr_write(struct file *file,
+					  const char __user *user_buffer,
+					  size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 1) ||
+	    (tmp > SMC_MAX_RMB) ||
+	    ((tmp * 10 * smc_ctrl_buf_cnt) > SMC_MAX_CQE)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_max_conn_per_lgr = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_max_conn_per_lgr_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_max_conn_per_lgr_show, PDE_DATA(inode));
+}
+
+static int smc_port_add_delay_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_port_add_delay);
+	return 0;
+}
+
+static ssize_t smc_port_add_delay_write(struct file *file,
+					const char __user *user_buffer,
+					size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 0) ||
+	    (tmp > SMC_MAX_PORT_ADD_DELAY)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_port_add_delay = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_port_add_delay_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_port_add_delay_show, PDE_DATA(inode));
+}
+
+static int smc_sndbuf_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_def_sndbuf_size);
+	return 0;
+}
+
+static ssize_t smc_sndbuf_write(struct file *file,
+				const char __user *user_buffer, size_t count,
+				loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < SK_MEM_QUANTUM) ||
+	    (tmp > sysctl_wmem_max)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_def_sndbuf_size = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_sndbuf_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_sndbuf_show, PDE_DATA(inode));
+}
+
+static int smc_rcvbuf_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_def_rcvbuf_size);
+	return 0;
+}
+
+static ssize_t smc_rcvbuf_write(struct file *file,
+				const char __user *user_buffer,
+				size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < SK_MEM_QUANTUM) ||
+	    (tmp > sysctl_rmem_max)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_def_rcvbuf_size = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_rcvbuf_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_rcvbuf_show, PDE_DATA(inode));
+}
+
+static const struct file_operations procfs_pnet_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_pnet_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_pnet_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_buff_cnt_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_buff_cnt_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_buff_cnt_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_max_conn_per_lgr_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_max_conn_per_lgr_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_max_conn_per_lgr_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_port_add_delay_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_port_add_delay_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_port_add_delay_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_sndbuf_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_sndbuf_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_sndbuf_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_rcvbuf_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_rcvbuf_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_rcvbuf_write,
+	.release = single_release,
+};
+
+int smc_proc_init(void)
+{
+	INIT_LIST_HEAD(&smc_pnet_list);
+	rwlock_init(&smc_pnet_list_lock);
+	procfs_root = proc_net_mkdir(&init_net, "smc", init_net.proc_net);
+	if (!procfs_root)
+		goto out;
+	procfs_pnet_conf =
+		proc_create_data("pnet_conf", S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_pnet_fops, NULL);
+	if (!procfs_pnet_conf)
+		goto out_smc;
+	procfs_buff_cnt =
+		proc_create_data("ctrl_buffer_count",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_buff_cnt_fops, NULL);
+	if (!procfs_buff_cnt)
+		goto out_pnet_conf;
+	procfs_max_conn_per_lgr =
+		proc_create_data("max_conn_per_lgr",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_max_conn_per_lgr_fops,
+				 NULL);
+	if (!procfs_max_conn_per_lgr)
+		goto out_ctrl_buffer_count;
+	procfs_port_add_delay =
+		proc_create_data("port_add_delay",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_port_add_delay_fops,
+				 NULL);
+	if (!procfs_port_add_delay)
+		goto out_max_conn_per_lgr;
+	procfs_sndbuf = proc_create_data("sndbuf", S_IFREG | S_IRUGO | S_IWUSR,
+					 procfs_root, &procfs_sndbuf_fops,
+					 NULL);
+	if (!procfs_sndbuf)
+		goto out_port_add_delay;
+	procfs_rcvbuf = proc_create_data("rcvbuf", S_IFREG | S_IRUGO | S_IWUSR,
+					 procfs_root, &procfs_rcvbuf_fops,
+					 NULL);
+	if (!procfs_rcvbuf)
+		goto out_sndbuf;
+	return 0;
+
+out_sndbuf:
+	remove_proc_entry("sndbuf", procfs_root);
+out_port_add_delay:
+	remove_proc_entry("port_add_delay", procfs_root);
+out_max_conn_per_lgr:
+	remove_proc_entry("max_conn_per_lgr", procfs_root);
+out_ctrl_buffer_count:
+	remove_proc_entry("ctrl_buffer_count", procfs_root);
+out_pnet_conf:
+	remove_proc_entry("pnet_conf", procfs_root);
+out_smc:
+	remove_proc_entry("smc", init_net.proc_net);
+out:
+	return -EFAULT;
+}
+
+void smc_proc_exit(void)
+{
+	struct smc_pnet_entry *entry, *tmp_entry;
+
+	write_lock(&smc_pnet_list_lock);
+	list_for_each_entry_safe(entry, tmp_entry, &smc_pnet_list, list) {
+		list_del(&entry->list);
+		kfree(entry);
+	}
+	write_unlock(&smc_pnet_list_lock);
+
+	if (procfs_rcvbuf)
+		remove_proc_entry("rcvbuf", procfs_root);
+	if (procfs_sndbuf)
+		remove_proc_entry("sndbuf", procfs_root);
+	if (procfs_port_add_delay)
+		remove_proc_entry("port_add_delay", procfs_root);
+	if (procfs_max_conn_per_lgr)
+		remove_proc_entry("max_conn_per_lgr", procfs_root);
+	if (procfs_buff_cnt)
+		remove_proc_entry("ctrl_buffer_count", procfs_root);
+	if (procfs_pnet_conf)
+		remove_proc_entry("pnet_conf", procfs_root);
+	if (procfs_root)
+		remove_proc_entry("smc", init_net.proc_net);
+}
-- 
2.3.8

Re: [PATCH V4 net-next 0/2] net: implement SMC-R solution

[David Miller]

From: Ursula Braun <ubraun@linux.vnet.ibm.com>
Date: Fri, 21 Aug 2015 13:30:27 +0200

> this is V4 of my SMC-R patches:
> Since you are asking for a solution "100% in our own separate module
> with our own can of worms", we have to give up the transparent detection
> whether a communication peer can do SMC-R or not (this has been the
> purpose of the rejected TCP hooks). Instead, we want just the new
> self-contained SMC-R socket family added to the kernel.
> By the way, since August 2015 the SMC-R Informational RFC is no longer
> a draft, but published as RFC7609.

The tcp_set_keepalive() layering hack is still beyond what I'm willing
to apply, sorry.

Also, as a coding style note, all of your code needs to be adjusted to
declare local function variables in "reverse christimas tree" order,
that is longest line to shortest.

[PATCH V2 net-next 2/3] net: introduce socket family constants

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

The new socket family is assigned the next available address / protocol
family constant 41.
Implementing SO_KEEPALIVE for SMC-R requires an extra hook in net/core/sock.c.

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 include/linux/socket.h |  4 +++-
 include/net/smc.h      | 13 +++++++++++++
 net/Kconfig            |  1 +
 net/Makefile           |  1 +
 net/core/sock.c        | 15 +++++++++++++--
 5 files changed, 31 insertions(+), 3 deletions(-)
 create mode 100644 include/net/smc.h

diff --git a/include/linux/socket.h b/include/linux/socket.h
index 5bf59c8..dd18d32 100644
--- a/include/linux/socket.h
+++ b/include/linux/socket.h
@@ -200,7 +200,8 @@ struct ucred {
 #define AF_ALG		38	/* Algorithm sockets		*/
 #define AF_NFC		39	/* NFC sockets			*/
 #define AF_VSOCK	40	/* vSockets			*/
-#define AF_MAX		41	/* For now.. */
+#define AF_SMC		41	/* smc sockets>->------->-------*/
+#define AF_MAX		42	/* For now.. */
 
 /* Protocol families, same as address families. */
 #define PF_UNSPEC	AF_UNSPEC
@@ -246,6 +247,7 @@ struct ucred {
 #define PF_ALG		AF_ALG
 #define PF_NFC		AF_NFC
 #define PF_VSOCK	AF_VSOCK
+#define PF_SMC		AF_SMC
 #define PF_MAX		AF_MAX
 
 /* Maximum queue length specifiable by listen.  */
diff --git a/include/net/smc.h b/include/net/smc.h
new file mode 100644
index 0000000..cd513ee
--- /dev/null
+++ b/include/net/smc.h
@@ -0,0 +1,13 @@
+/*
+ * SMC		Definitions for the SMC protocol.
+ *
+ * Author:	Ursula Braun <ursula.braun@de.ibm.com>
+ */
+#ifndef _SMC_H
+#define _SMC_H
+
+/* SMC socket options - disjunct with TCP socket options */
+#define SMC_KEEPALIVE		99	/* start/stop keepalives */
+
+#endif /* _SMC_H */
+
diff --git a/net/Kconfig b/net/Kconfig
index 57a7c5a..d829694 100644
--- a/net/Kconfig
+++ b/net/Kconfig
@@ -54,6 +54,7 @@ source "net/packet/Kconfig"
 source "net/unix/Kconfig"
 source "net/xfrm/Kconfig"
 source "net/iucv/Kconfig"
+source "net/smc/Kconfig"
 
 config INET
 	bool "TCP/IP networking"
diff --git a/net/Makefile b/net/Makefile
index 3995613..33a5f54 100644
--- a/net/Makefile
+++ b/net/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_MAC80211)		+= mac80211/
 obj-$(CONFIG_TIPC)		+= tipc/
 obj-$(CONFIG_NETLABEL)		+= netlabel/
 obj-$(CONFIG_IUCV)		+= iucv/
+obj-$(CONFIG_AFSMC)		+= smc/
 obj-$(CONFIG_RFKILL)		+= rfkill/
 obj-$(CONFIG_NET_9P)		+= 9p/
 obj-$(CONFIG_CAIF)		+= caif/
diff --git a/net/core/sock.c b/net/core/sock.c
index 08f16db..5608a70 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -139,6 +139,9 @@
 
 #ifdef CONFIG_INET
 #include <net/tcp.h>
+#if IS_ENABLED(CONFIG_AFSMC)
+#include <net/smc.h>
+#endif /* CONFIG_AFSMC */
 #endif
 
 #include <net/busy_poll.h>
@@ -784,8 +787,16 @@ set_rcvbuf:
 	case SO_KEEPALIVE:
 #ifdef CONFIG_INET
 		if (sk->sk_protocol == IPPROTO_TCP &&
-		    sk->sk_type == SOCK_STREAM)
-			tcp_set_keepalive(sk, valbool);
+		    sk->sk_type == SOCK_STREAM) {
+#if IS_ENABLED(CONFIG_AFSMC)
+			if (sk->sk_family == AF_SMC)
+				sock->ops->setsockopt(sock, SOL_TCP,
+						      SMC_KEEPALIVE,
+						      optval, optlen);
+			else
+#endif
+				tcp_set_keepalive(sk, valbool);
+		}
 #endif
 		sock_valbool_flag(sk, SOCK_KEEPOPEN, valbool);
 		break;
-- 
2.3.8

Re: [PATCH V2 net-next 2/3] net: introduce socket family constants

[David Miller]

From: Ursula Braun <ubraun@linux.vnet.ibm.com>
Date: Tue, 14 Jul 2015 14:42:34 +0200

> diff --git a/net/Kconfig b/net/Kconfig
> index 57a7c5a..d829694 100644
> --- a/net/Kconfig
> +++ b/net/Kconfig
> @@ -54,6 +54,7 @@ source "net/packet/Kconfig"
>  source "net/unix/Kconfig"
>  source "net/xfrm/Kconfig"
>  source "net/iucv/Kconfig"
> +source "net/smc/Kconfig"

This breaks the build, every patch in the series must leave the kernel
in a consistent, building, working state.

This patch series submission is of very low quality.

[PATCH V2 net-next 3/3] smc: introduce socket family AF_SMC

[Ursula Braun]

From: Ursula Braun <ursula.braun@de.ibm.com>

In 2013, IBM introduced an optimized communications solution for the
IBM zEnterprise EC12 and BC12 (s390 in Linux terminology) that is
comprised of the IBM 10GbE RoCE Express feature with Shared Memory
Communications-RDMA (SMC-R) protocol [1].
SMC-R is designed for the enterprise data center environment and is an open
protocol as specified in the informational RFC [2]. The final draft
submitted by IBM has been approved for publication and is in the final
editorial stage. Another implementation of this protocol is available since
2013 with IBM z/OS Version 2 Release 1. 

SMC-R provides a “sockets over RDMA” solution that leverages industry
standard RDMA over Converged Ethernet (RoCE) technology.

IBM has developed a Linux implementation of the SMC-R standard. A new
socket protocol family AF_SMC is introduced. A preload library can be used
to enable TCP-based applications to use SMC-R without changes. 

Key aspects of SMC-R are: 
1. Provides optimized performance compared to standard TCP/IP over Ethernet
   within the data center for both request/response (latency) and streaming
   workloads (CPU savings) [3]. 
2. In order to preserve the traditional network administrative model the
   SMC-R protocol ties into the existing IP addresses and uses TCP's
   handshake to establish connections. This allows existing management
   tools and security infrastructure to control the creation of SMC
   connections.
3. The SMC-R protocol logically bonds multiple RoCE adapters together
   providing redundancy with transparent fail-over for improved high
   availability, increased bandwidth and load balancing across multiple
   RDMA-capable devices.
4. Due to its handshake protocol, SMC-R is compatible with (transparent to)
   existing TCP connection load balancers that are commonly used in the
   enterprise data center environment for multi-tier application workloads.
5. SMC-R's handshake protocol allows for transparent fallback to TCP/IP,
   should one of the peers not be capable of the protocol.

Additional SMC-R overview and reference materials are available [1].  

The SMC-R “rendezvous" protocol eliminates the need for RDMA-CM and the
exchange occurs through an initial TCP connection. Building on a TCP
connection to establish an SMC-R connection solves many key requirements,
including #4 and #5 above.
The rendezvous process occurs in 2 phases: 
1. TCP/IP 3-way exchange:
   Initiated when both client and server indicate SMC-R capability by
   including TCP experimental options on the TCP/IP 3-way handshake (syn
   flows) as    described in RFC6994 [4]. The ExID assigned by IANA is
   0xE2D4C3D9 [5]. 
2. SMC-R 3-way exchange:
   When both partners indicate SMC-R capability then at the completion of
   the 3-way TCP handshake the SMC-R layers in each peer take control of
   the TCP connection and exchange their RDMA credentials. If this 3-way
   exchange completes successfully the connection continues using SMC-R.
   If the exchange is not successful the connections falls back to standard
   TCP/IP. 

A connection using SMC-R starts out setting up a connection using the
TCP/IP protocol [2]. Thus an internal smc-managed TCP socket is
established, whose traffic flows across an interface belonging to the same
Converged Ethernet fabric, defined by a so-called "pnet table" [1].

As part of the connection setup, flags indicating the support
for SMC-R are used to negotiate SMC-R usage. If any of the two parties
involved does not support SMC-R, the connection falls back to plain TCP/IP
usage (the previously established TCP connection will be used for any
further traffic). If both parties support SMC-R, the TCP/IP connection is
used to exchange the necessary connection data for setup of RoCE RC-QP
connection in the so-called "rendezvous handshake".

The code also implements also the link management of the SMC-R protocol
forming link groups for each remote host and provide for
- dynamic addition of further RoCE Host Channel Adapters (HCAs) to form a
  so-called link group
- seamless failover, moving existing connections from a failed HCA
  to the remaining HCAs in a link group
- load balancing of connections across the links in a link group

SMC-R design defines memory areas for data transport through RDMA called
RDMA Memory Buffer (RMB) which are split up into several RMB Elements
(RMBEs) assigned to single connections. This initial Linux implementation
requires real contiguous storage for these RMBs and uses a separate RMB
with just 1 RMBE for every connection. Its size is derived from:
- a system wide default minimum SMC receive buffer size (tailorable by
  /proc/net/smc/rcvbuf
- if given, the socket specific SETSOCKOPT SO_RCVBUF configuration
- the size of the tcp socket receive buffer determined from the system wide
  tcp socket receive buffer configuration
- the amount of available contiguous storage
In the Linux implementation, an equivalent concept exists for the socket
send buffers [1].

In the future, SMC-R will be enhanced to cover:
- IPv6 support
- Tracing
- Statistic hooks

References:
[1] SMC-R Overview and Reference Materials:
    http://www-01.ibm.com/software/network/commserver/SMCR/ 
[2] SMC-R Informational RFC:
    http://tools.ietf.org/html/draft-fox-tcpm-shared-memory-rdma-07
[3] Linux SMC-R Overview and Performance Summary
    (archs x86 and s390):
    http://www-01.ibm.com/software/network/commserver/SMCR/ 
[4] Shared Use of TCP Experimental Options RFC 6994:
    https://tools.ietf.org/rfc/rfc6994.txt    
[5] IANA ExID SMCR: 
    http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xhtml#tcp-exids

Signed-off-by: Ursula Braun <ursula.braun@de.ibm.com>
---
 net/smc/Kconfig    |    9 +
 net/smc/Makefile   |    3 +
 net/smc/af_smc.c   | 3142 +++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/af_smc.h   |  706 +++++++++++
 net/smc/smc_core.c | 3291 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_llc.c  | 1597 +++++++++++++++++++++++++
 net/smc/smc_llc.h  |  192 +++
 net/smc/smc_proc.c |  884 ++++++++++++++
 8 files changed, 9824 insertions(+)
 create mode 100644 net/smc/Kconfig
 create mode 100644 net/smc/Makefile
 create mode 100644 net/smc/af_smc.c
 create mode 100644 net/smc/af_smc.h
 create mode 100644 net/smc/smc_core.c
 create mode 100644 net/smc/smc_llc.c
 create mode 100644 net/smc/smc_llc.h
 create mode 100644 net/smc/smc_proc.c

diff --git a/net/smc/Kconfig b/net/smc/Kconfig
new file mode 100644
index 0000000..4ae42d5
--- /dev/null
+++ b/net/smc/Kconfig
@@ -0,0 +1,9 @@
+config AFSMC
+	tristate "AF_SMC socket address family"
+	depends on INET && INFINIBAND
+	---help---
+	  SMC_R provides a "sockets over RDMA" solution making use of
+	  RDMA over Converged Ethernet (ROCE) technology.
+	  Socket family AF_SMC implements this solution for Linux.
+
+	  Select this option if you want to run AF_SMC socket applications
diff --git a/net/smc/Makefile b/net/smc/Makefile
new file mode 100644
index 0000000..76017f0
--- /dev/null
+++ b/net/smc/Makefile
@@ -0,0 +1,3 @@
+ccflags-y += -I$(src)
+smc-y := af_smc.o smc_core.o smc_proc.o smc_llc.o
+obj-$(CONFIG_AFSMC)	+= smc.o
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
new file mode 100644
index 0000000..0b2f26e
--- /dev/null
+++ b/net/smc/af_smc.c
@@ -0,0 +1,3142 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  AF_SMC protocol family socket handler
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              Stefan Raspl <raspl@linux.vnet.ibm.com>
+ *              based on prototype from Frank Blaschka
+ */
+
+#define KMSG_COMPONENT "af_smc"
+#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/seq_file.h>
+#include <linux/uaccess.h>
+#include <linux/workqueue.h>
+#include <linux/proc_fs.h>
+#include <linux/types.h>
+#include <net/sock.h>
+#include <net/inet_sock.h>
+#include <net/tcp.h>
+#include <net/smc.h>
+#include <asm/byteorder.h>
+#include <asm/ioctls.h>
+#include <rdma/ib_verbs.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+#define VERSION "1.0"
+
+static struct proto smc_proto = {
+	.name		= "AF_SMC",
+	.owner		= THIS_MODULE,
+	.obj_size	= sizeof(struct smc_sock),
+};
+
+static struct sock *smc_sock_alloc(struct socket *, int, gfp_t);
+static void smc_tcp_listen_worker(struct work_struct *);
+static int smc_getsockopt_from_tcp(struct smc_sock *, int);
+static struct sock *smc_accept_dequeue(struct sock *, struct socket *);
+
+struct list_head smc_ib_devices;	/* list of smc_ib_devices */
+static spinlock_t smc_ib_dev_lock;	/* protect smc_ib_devices_list */
+struct smc_lgr_list_struct smc_lgr_list;	/* list of link_groups */
+struct list_head smc_lgr_freeing_list;	/* list of link groups to be freed */
+static u8 local_peerid[8] = SMC_LOCAL_PEERID_RESET;	/* unique system
+							 * identifier
+							 */
+u32 smc_ctrl_buf_cnt = 16;		/* # of control buffers per link */
+u32 smc_max_conn_per_lgr = 200;		/* max. # of connections per lgr */
+u32 smc_port_add_delay;		/* port add delay (for certain switches) */
+atomic_t smc_lgr_num = ATOMIC_INIT(0);	/* unique link group number */
+unsigned int smc_def_rcvbuf_size = 65532;	/* def. receive buffer size */
+unsigned int smc_def_sndbuf_size = 65532;	/* def. send buffer size */
+
+atomic_t smc_reconfiguring = ATOMIC_INIT(0);	/* serialize port_add/port_err
+						 * and free_lgr
+						 */
+
+#ifdef CONFIG_S390
+#include <asm/ebcdic.h>
+#endif
+
+/* link is up - establish alternate link if applicable
+ * must be called with lgr->conf_mutex taken
+ */
+void smc_link_up(struct smc_link_group *lgr)
+{
+	struct smc_link *link = NULL;
+	struct smc_roce_defs rocdefs;
+	int i;
+	u8 p1, p2;
+
+	/* determine working link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			link = &lgr->lnk[i];
+			break;
+		}
+	}
+	if (!link || (lgr->lgr_type == SYMMETRIC) ||
+	    (lgr->lgr_type == ASYMMETRIC_PEER))
+		return;
+again:
+	if (!lgr->port_up)
+		return;
+	p1 = lgr->port_up & 1;
+	p2 = lgr->port_up & 2;
+	if (lgr->role == SMC_SERV) {
+		struct llc_add_link_fake_msg add_llc;
+
+		memset(&add_llc, 0, sizeof(add_llc));
+		add_llc.hd.length = sizeof(struct llc_msg);
+		add_llc.hd.type = LLC_ADD_LINK;
+		add_llc.ibdev = lgr->ibdev_up;
+		add_llc.port_num = p1 ? p1 : p2;
+		add_llc.hd.flags |= LLC_FLAG_LOCAL;
+		llc_enqueue(link, (struct llc_msg *)&add_llc);
+		lgr->port_up &= ~add_llc.port_num;
+	} else {
+		/* invite server to start ADD LINK */
+		rocdefs.ibdev = lgr->ibdev_up;
+		rocdefs.port = p1 ? p1 : p2;
+		rocdefs.vlan = lgr->vlan;
+		smc_gid_by_dev(&rocdefs);
+		if (lgr->llc_ctl[LLC_SERV].active != LLC_GRP_NONE) {
+			mutex_unlock(&lgr->conf_mutex);
+			wait_event_interruptible_timeout(lgr->llc_waiter,
+				(lgr->llc_ctl[LLC_SERV].active == LLC_GRP_NONE),
+				LLC_WAIT_TIMEO);
+			mutex_lock(&lgr->conf_mutex);
+		}
+		if (!list_empty(&lgr->list)) {
+			if (smc_port_active(rocdefs.ibdev, rocdefs.port))
+				llc_send_add_link(link, &rocdefs, NULL, LLC_REQ,
+						  LLC_NO_NOTIFY);
+		}
+		lgr->port_up &= ~rocdefs.port;
+	}
+	goto again;
+}
+
+static void smc_port_add_worker(struct work_struct *work)
+{
+	struct smc_ib_device *smc_ibdev =
+				container_of(work, struct smc_ib_device,
+					     port_add_work.work);
+	struct smc_link_group *lgr;
+	u8 port_num = 0;
+	int i;
+
+	atomic_inc(&smc_reconfiguring);
+	for (i = 0; i < 2; i++) {
+		char pnet[SMC_MAX_PNET_ID_LEN];
+		int rc;
+
+		if (!smc_ibdev->port_add[i])
+			continue;
+		smc_check_port_attr(smc_ibdev, i);
+		if (smc_ibdev->pattr[i].state != IB_PORT_ACTIVE)
+			continue;
+		port_num = i + 1;
+		memset(pnet, 0, sizeof(pnet));
+		rc = smc_pnet_by_ibdev(port_num, pnet, smc_ibdev);
+		if (rc)
+			continue;
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+			if (strncmp(pnet, lgr->pnet_id, sizeof(pnet)) ||
+			    (lgr->lgr_type == SYMMETRIC) ||
+			    (lgr->lgr_type == ASYMMETRIC_PEER))
+				continue;
+			if (cmpxchg(&lgr->lnk_up, 0, 1) == 0) {
+				lgr->ibdev_up = smc_ibdev;
+				lgr->port_up |= port_num;
+				schedule_work(&lgr->link_up_work);
+			}
+		}
+		spin_unlock_bh(&smc_lgr_list.lock);
+		smc_ibdev->port_add[i] = 0;
+	}
+	atomic_dec(&smc_reconfiguring);
+}
+
+/* link is down - switch connections to alternate link
+ * must be called with lgr->conf_mutex taken
+ */
+void smc_link_down(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct smc_link *link = &lgr->lnk[lnk_idx];
+	struct smc_ib_device *smc_ibdev = link->roce_defs.ibdev;
+	int to_lnk_idx;
+	u8 port_num = link->roce_defs.port;
+
+	smc_modify_qp_reset(link);
+	to_lnk_idx = smc_switch_conns(lgr, lnk_idx, 1);
+	if (to_lnk_idx < 0) { /* no backup link available */
+		smc_free_link(lgr, lnk_idx);
+		return;
+	}
+	lgr->lgr_type = SINGLE;
+	if (lgr->role == SMC_SERV) {
+		struct llc_del_link_msg del_llc;
+
+		memset(&del_llc, 0, sizeof(del_llc));
+		del_llc.hd.length = sizeof(struct llc_msg);
+		del_llc.hd.type = LLC_DEL_LINK;
+		del_llc.link_num = link->link_id;
+		del_llc.reason = htonl(LLC_DEL_LOST_PATH);
+		del_llc.hd.flags |= LLC_FLAG_DEL_ORDERLY;
+		del_llc.hd.flags |= LLC_FLAG_LOCAL;
+		llc_enqueue(&lgr->lnk[to_lnk_idx], (struct llc_msg *)&del_llc);
+		/* try to restart the link */
+		if (!list_empty(&lgr->list) &&
+		    !list_empty(&smc_ibdev->list) &&
+		    smc_port_active(smc_ibdev, port_num)) {
+			lgr->ibdev_up = smc_ibdev;
+			lgr->port_up |= port_num;
+			schedule_work(&lgr->link_up_work);
+		}
+	} else {
+		if (lgr->llc_ctl[LLC_SERV].active != LLC_GRP_NONE) {
+			mutex_unlock(&lgr->conf_mutex);
+			wait_event_interruptible_timeout(lgr->llc_waiter,
+				(lgr->llc_ctl[LLC_SERV].active ==
+							LLC_GRP_NONE),
+				LLC_WAIT_TIMEO);
+			mutex_lock(&lgr->conf_mutex);
+		}
+		llc_send_del_link(&lgr->lnk[to_lnk_idx], link,
+				  LLC_FLAG_DEL_ORDERLY,
+				  LLC_DEL_LOST_PATH, LLC_NO_NOTIFY);
+	}
+}
+
+static void smc_port_err_worker(struct work_struct *work)
+{
+	struct smc_ib_device *smc_ibdev =
+		container_of(work, struct smc_ib_device, port_err_work);
+	struct smc_link_group *lgr;
+	int i, rc1, rc2;
+	u8 port_num;
+	char pnet1[SMC_MAX_PNET_ID_LEN], pnet2[SMC_MAX_PNET_ID_LEN];
+
+	atomic_inc(&smc_reconfiguring);
+	for (i = 0; i < 2; i++) {
+		if (!smc_ibdev->port_err[i])
+			continue;
+		smc_check_port_attr(smc_ibdev, i);
+		port_num = i + 1;
+		rc1 = smc_pnet_by_ibdev(1, pnet1, smc_ibdev);
+		rc2 = smc_pnet_by_ibdev(2, pnet2, smc_ibdev);
+		smc_ibdev->port_err[i] = 0;
+		if (rc1 && rc2) /* no pnet definition for this device */
+			continue;
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+			int j;
+
+			if (strncmp(pnet1, lgr->pnet_id, sizeof(pnet1)) &&
+			    strncmp(pnet2, lgr->pnet_id, sizeof(pnet2)))
+				continue;
+			if (lgr->lgr_type == NONE) /* no links */
+				continue;
+			for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+				struct smc_link *lnk = &lgr->lnk[j];
+
+				if ((lnk->roce_defs.ibdev == smc_ibdev) &&
+				    (lnk->roce_defs.port == port_num) &&
+				    cmpxchg(&lgr->lnk_down[j], 0, 1) == 0)
+					schedule_work(&lgr->link_down_work);
+			}
+		}
+		spin_unlock_bh(&smc_lgr_list.lock);
+	}
+	atomic_dec(&smc_reconfiguring);
+}
+
+static void smc_global_event_handler(struct ib_event_handler *handler,
+				     struct ib_event *event)
+{
+	struct smc_ib_device *dev;
+	u8 port_num = event->element.port_num - 1;
+
+	/* irq context! */
+	dev = container_of(handler, struct smc_ib_device, event_handler);
+	switch (event->event) {
+	case IB_EVENT_DEVICE_FATAL:
+		dev->port_err[0] = 1;
+		dev->port_err[1] = 1;
+		if ((cmpxchg(&dev->port_err[0], 0, 1) == 0) ||
+		    (cmpxchg(&dev->port_err[1], 0, 1) == 0))
+			schedule_work(&dev->port_err_work);
+		break;
+	case IB_EVENT_PORT_ERR:
+		if (cmpxchg(&dev->port_err[port_num], 0, 1) == 0)
+			schedule_work(&dev->port_err_work);
+		break;
+	case IB_EVENT_PORT_ACTIVE:
+		if (cmpxchg(&dev->port_add[port_num], 0, 1) == 0)
+			schedule_delayed_work(&dev->port_add_work,
+					      smc_port_add_delay);
+		break;
+	case IB_EVENT_GID_CHANGE:
+		break;
+	default:
+		break;
+	}
+}
+
+static void smc_ib_add_dev(struct ib_device *dev)
+{
+	struct smc_ib_device *smc_ibdev;
+	struct smc_roce_defs rocdefs;
+	struct ib_cq_init_attr attr = {.cqe = SMC_MAX_CQE, .comp_vector = 0};
+
+	if (dev->node_type != RDMA_NODE_IB_CA)
+		return;
+	smc_ibdev = kzalloc(sizeof(*smc_ibdev), GFP_KERNEL);
+	if (!smc_ibdev)
+		return;
+	smc_ibdev->dev = dev;
+	smc_ibdev->roce_cq_send = ib_create_cq(dev, smc_cq_handler_send,
+		smc_cq_event_handler, smc_ibdev, &attr);
+	if (IS_ERR(smc_ibdev->roce_cq_send))
+		goto err;
+	smc_ibdev->roce_cq_recv = ib_create_cq(dev, smc_cq_handler_recv,
+		smc_cq_event_handler, smc_ibdev, &attr);
+	if (IS_ERR(smc_ibdev->roce_cq_recv))
+		goto err;
+	if (!strncmp(local_peerid, SMC_LOCAL_PEERID_RESET,
+		     sizeof(local_peerid))) {
+		rocdefs.ibdev = smc_ibdev;
+		rocdefs.port = 1;
+		smc_get_ib_mac(&rocdefs);
+		memcpy(&local_peerid[2], &rocdefs.mac, sizeof(rocdefs.mac));
+		get_random_bytes_arch(&local_peerid[0], 2);
+	}
+	spin_lock_bh(&smc_ib_dev_lock);
+	list_add_tail(&smc_ibdev->list, &smc_ib_devices);
+	spin_unlock_bh(&smc_ib_dev_lock);
+	ib_set_client_data(dev, &smc_ib_client, smc_ibdev);
+	smc_check_dev_attr(smc_ibdev);
+	INIT_WORK(&smc_ibdev->port_err_work, smc_port_err_worker);
+	INIT_DELAYED_WORK(&smc_ibdev->port_add_work, smc_port_add_worker);
+	INIT_IB_EVENT_HANDLER(&smc_ibdev->event_handler, dev,
+			      smc_global_event_handler);
+	ib_register_event_handler(&smc_ibdev->event_handler);
+	tasklet_init(&smc_ibdev->recv_tasklet, smc_recv_tasklet_fn,
+		     (unsigned long)smc_ibdev);
+	tasklet_init(&smc_ibdev->send_tasklet, smc_send_tasklet_fn,
+		     (unsigned long)smc_ibdev);
+	return;
+
+err:
+	kfree(smc_ibdev);
+}
+
+static void smc_wait_for_freed_links(struct smc_ib_device *sdev)
+{
+	struct smc_link_group *lgr;
+	struct smc_link *lnk;
+	int i;
+
+again:
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			lnk = &lgr->lnk[i];
+			if (lnk->roce_defs.ibdev == sdev) {
+				spin_unlock_bh(&smc_lgr_list.lock);
+				switch (atomic_read(&lnk->state)) {
+				case SMC_LINK_FREED:
+				case SMC_LINK_UP: /* wait for port_err finish */
+					msleep(LLC_WAIT_FIRST_TIMEO);
+					break;
+				case SMC_LINK_DOWN:
+					smc_free_link(lgr, i);
+					break;
+				default:
+					break;
+				}
+				goto again;
+			}
+		}
+	}
+	list_for_each_entry(lgr, &smc_lgr_freeing_list, list) {
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			lnk = &lgr->lnk[i];
+			if (lnk->roce_defs.ibdev == sdev) {
+				spin_unlock_bh(&smc_lgr_list.lock);
+				switch (atomic_read(&lnk->state)) {
+				case SMC_LINK_FREED:
+					msleep(LLC_WAIT_FIRST_TIMEO);
+					break;
+				case SMC_LINK_UP:
+					atomic_set(&lnk->state, SMC_LINK_DOWN);
+					/* fall through */
+				case SMC_LINK_DOWN:
+					smc_free_link(lgr, i);
+					break;
+				default:
+					break;
+				}
+				goto again;
+			}
+		}
+	}
+	spin_unlock_bh(&smc_lgr_list.lock);
+}
+
+static void smc_ib_remove_dev(struct ib_device *dev)
+{
+	struct smc_ib_device *smc_ibdev;
+
+	atomic_inc(&smc_reconfiguring);
+	smc_ibdev = ib_get_client_data(dev, &smc_ib_client);
+	ib_unregister_event_handler(&smc_ibdev->event_handler);
+	smc_ibdev->port_add[0] = 0;
+	smc_ibdev->port_add[1] = 0;
+	smc_ibdev->port_err[0] = 1;
+	smc_ibdev->port_err[1] = 1;
+	schedule_work(&smc_ibdev->port_err_work);
+	ib_set_client_data(dev, &smc_ib_client, NULL);
+	spin_lock_bh(&smc_ib_dev_lock);
+	list_del_init(&smc_ibdev->list);
+	spin_unlock_bh(&smc_ib_dev_lock);
+	smc_wait_for_freed_links(smc_ibdev);
+	tasklet_kill(&smc_ibdev->recv_tasklet);
+	tasklet_kill(&smc_ibdev->send_tasklet);
+	ib_destroy_cq(smc_ibdev->roce_cq_send);
+	ib_destroy_cq(smc_ibdev->roce_cq_recv);
+	flush_work(&smc_ibdev->port_err_work);
+	kfree(smc_ibdev);
+	atomic_dec(&smc_reconfiguring);
+}
+
+struct ib_client smc_ib_client = {
+	.name	= "smc_ib",
+	.add	= smc_ib_add_dev,
+	.remove = smc_ib_remove_dev
+};
+
+void smc_sock_wake_rx(struct sock *sk)
+{
+	struct socket_wq *wq;
+
+	/* derived from sock_def_readable() */
+	/* Attention: sk_data_ready=smc_sock_wake_rx called in listen_worker */
+	rcu_read_lock();
+	wq = rcu_dereference(sk->sk_wq);
+	if (wq_has_sleeper(wq)) {
+		wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLPRI |
+						POLLRDNORM | POLLRDBAND);
+	}
+	if ((sk->sk_shutdown == SHUTDOWN_MASK) ||
+	    (sk->sk_state == SMC_CLOSED))
+		sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_HUP);
+	else
+		sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
+	rcu_read_unlock();
+}
+
+void smc_sock_wake_tx(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct socket *sock = sk->sk_socket;
+	struct socket_wq *wq;
+
+/* similar to sk_stream_write_space */
+	if (atomic_read(&smc->conn.tx_buf_space) && sock) {
+		clear_bit(SOCK_NOSPACE, &sock->flags);
+		rcu_read_lock();
+		wq = rcu_dereference(sk->sk_wq);
+		if (wq_has_sleeper(wq)) {
+			wake_up_interruptible_poll(&wq->wait,
+						   POLLOUT | POLLWRNORM |
+						   POLLWRBAND);
+		}
+		if (wq && wq->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN))
+			sock_wake_async(sock, SOCK_WAKE_SPACE, POLL_OUT);
+		rcu_read_unlock();
+	}
+}
+
+static void smc_sk_release(struct smc_sock *, int);
+static void smc_sock_destruct(struct sock *sk);
+
+static void smc_destruct_non_accepted(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int i;
+
+	sock_hold(sk);
+	lock_sock(sk);
+	smc_sk_release(smc, 0);
+	if (smc->tcpsocket) {
+		struct socket *tcp;
+
+		tcp = smc->tcpsocket;
+		smc->tcpsocket = NULL;
+		sock_release(tcp);
+	}
+	release_sock(sk);
+	if (!smc->use_tcp) {
+		for (i = 0;
+		     ((i < 10) &&
+		      (smc_pending_sends(conn) ||
+		       !smc_close_received(conn)));
+		     i++) {
+			msleep(SMC_WAIT_PENDING_SENDS_TIMEO);
+		}
+	}
+	if (cmpxchg(&smc->sock_put_done, 0, 1) == 0) {
+		sock_set_flag(sk, SOCK_ZAPPED);
+		sock_set_flag(sk, SOCK_DEAD);
+		schedule_work(&smc->release_work);
+	}
+	sock_put(sk);
+}
+
+static void smc_sock_cleanup_listen(struct sock *parent)
+{
+	struct sock *sk;
+
+	/* Close non-accepted connections */
+	while ((sk = smc_accept_dequeue(parent, NULL)))
+		smc_destruct_non_accepted(sk);
+}
+
+static void smc_wait_close(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	signed long timeout;
+	DEFINE_WAIT(wait);
+
+	/* cancel CORK */
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+		schedule_delayed_work(&smc->write_work, 0);
+	}
+	timeout = sk->sk_lingertime ? sk->sk_lingertime : MAX_SCHEDULE_TIMEOUT;
+	if (smc_prepared_sends(&smc->conn) && !sk->sk_err &&
+	    !(current->flags & PF_EXITING)) {
+		release_sock(sk);
+		wait_event_interruptible_timeout(smc->prep_waiter,
+						!smc_prepared_sends(&smc->conn),
+						timeout);
+		lock_sock(sk);
+	}
+	timeout = SMC_WAIT_PENDING_SENDS_TIMEO;
+	if (smc_pending_sends(&smc->conn) &&
+	    !(current->flags & PF_EXITING)) {
+		set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+		do {
+			prepare_to_wait(sk_sleep(sk), &wait,
+					TASK_INTERRUPTIBLE);
+			if (sk_wait_event(sk, &timeout,
+					  !smc_pending_sends(&smc->conn)))
+				break;
+		} while (!signal_pending(current) && timeout);
+		clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+		finish_wait(sk_sleep(sk), &wait);
+	}
+}
+
+static int smc_wait_for_pending_sends(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	u32 i, j;
+	int rc;
+
+	release_sock(&smc->sk);
+	rc = wait_event_timeout(smc->destruct_waiter,
+				!smc_pending_sends(&smc->conn),
+				SMC_WAIT_PENDING_SENDS_TIMEO);
+	lock_sock(&smc->sk);
+	if (!rc) { /* timeout reached */
+		for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+			link = &smc->conn.lgr->lnk[j];
+			if (!link->link_id)
+				continue;
+			for (i = 0; i < link->send_wr_num; i++)
+				if (link->pending[i].conn == &smc->conn) {
+					link->pending[i].conn = NULL;
+					clear_bit(i, link->pending_bit_mask);
+				}
+		}
+	}
+	return rc;
+}
+
+static void smc_conn_term_abnormal(struct smc_sock *smc, int old_state)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+
+	sk_stop_timer(sk, &sk->sk_timer);
+	while (work_busy(&smc->timer_work))
+		msleep(20);
+	if (sk->sk_state == SMC_PEERCLW2 ||
+	    sk->sk_state == SMC_PEERFINCLW ||
+	    sk->sk_state == SMC_CLOSED ||
+	    sk->sk_state == SMC_INIT ||
+	    sk->sk_state == SMC_PEERABORTW) {
+		sk->sk_state = SMC_CLOSED;
+		if (timer_pending(&smc->fin_timer))
+			mod_timer_pending(&smc->fin_timer, jiffies);
+	} else {
+		if (sk->sk_state == SMC_PEERCLW1) {
+			/* enable wait_close wakeup */
+			smc_clear_pending_sends(conn);
+		}
+		/* cancel close waiting */
+		atomic64_set(&conn->tx_curs_prep.s.acurs,
+			     conn->tx_curs_sent.s.lcurs);
+		atomic64_set(&conn->tx_curs_fin.s.acurs,
+			     conn->tx_curs_sent.s.lcurs);
+		sk->sk_state = SMC_PROCESSABORT;
+		/* purge all in-flight data, do abnormal term step */
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+		if ((old_state != SMC_PROCESSABORT) && (conn->lgr) &&
+		    (conn->lgr->lgr_type != NONE)) {
+			smc_ctrl_send(conn, conn->lnk_idx);
+			if (conn->lgr->lgr_type != SINGLE)
+				smc_send_close_alt(smc);
+		}
+		if (sock_flag(sk, SOCK_DEAD))
+			mod_timer_pending(&smc->fin_timer,
+					  jiffies + TCP_TIMEWAIT_LEN);
+	}
+	sk->sk_err = ECONNABORTED;
+}
+
+static void smc_accept_unlink(struct sock *);
+
+void smc_conn_term(struct smc_sock *smc)
+{
+	struct sock *tcpsk;
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int old_state;
+
+	old_state = sk->sk_state;
+	/* Handle abnormal termination */
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		smc_conn_term_abnormal(smc, old_state);
+		goto out;
+	}
+	switch (sk->sk_state) {
+	/* Normal termination - Passive close part */
+	case SMC_INIT:
+	case SMC_ACTIVE:
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done ||
+		    conn->local_rx_ctrl.conn_state_flags.closed_conn) {
+			/* complete any outstanding recv with zero-length
+			 * if peerclosedconn and pending data to be written
+			 * then reset conn
+			 */
+			sk->sk_state = SMC_APPLCLW1;
+			if (sock_flag(sk, SOCK_DEAD))
+				mod_timer(&smc->fin_timer,
+					  jiffies + TCP_TIMEWAIT_LEN);
+		}
+		break;
+	case SMC_PEERFINCLW:
+		if (conn->local_rx_ctrl.conn_state_flags.closed_conn)
+			sk->sk_state = SMC_CLOSED;
+		break;
+		/* Normal termination - Active close part */
+	case SMC_PEERCLW1:
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done) {
+			/* complete any outstanding recv with zero-length */
+			sk->sk_state = SMC_PEERCLW2;
+		} /* fall through */
+	case SMC_PEERCLW2:
+		if (conn->local_rx_ctrl.conn_state_flags.closed_conn) {
+			struct smc_e_ctrl *tx_ctrl = &conn->local_tx_ctrl;
+
+			/* complete any outstanding recv with zero-length */
+			if (sk->sk_shutdown == SHUTDOWN_MASK &&
+			    (tx_ctrl->conn_state_flags.closed_conn ||
+			     tx_ctrl->conn_state_flags.abnormal_close)) {
+				sk->sk_state = SMC_CLOSED;
+				if (timer_pending(&smc->fin_timer))
+					mod_timer_pending(&smc->fin_timer,
+							  jiffies);
+			} else {
+				sk->sk_state = SMC_APPLFINCLW;
+			}
+		}
+		break;
+	default:
+		break;
+	}
+out:
+	if ((sk->sk_err == ECONNABORTED) &&
+	    !list_empty(&smc->accept_q))
+		smc_accept_unlink(sk);
+	if (smc_stop_received(conn)) {
+		sk->sk_shutdown = sk->sk_shutdown | RCV_SHUTDOWN;
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			tcpsk = smc->tcpsocket->sk;
+			tcpsk->sk_shutdown = tcpsk->sk_shutdown | RCV_SHUTDOWN;
+		}
+	}
+	if (smc_close_received(conn) &&
+	    (sk->sk_state == SMC_CLOSED) &&
+	    !sk->sk_socket &&
+	    !smc_pending_sends(conn))
+		sock_set_flag(sk, SOCK_DEAD);
+	if ((old_state != sk->sk_state) &&
+	    (old_state != SMC_INIT))
+		sk->sk_state_change(sk);
+}
+
+static void smc_sk_release(struct smc_sock *smc, int wait)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+
+	if (smc->use_tcp) {
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		goto out;
+	}
+	sk_stop_timer(sk, &sk->sk_timer);
+	cancel_work_sync(&smc->timer_work);
+
+	sk->sk_shutdown = SHUTDOWN_MASK;
+
+	switch (sk->sk_state) {
+	case SMC_INIT:
+	case SMC_CLOSED:
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		break;
+	case SMC_LISTEN:
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			rc = kernel_sock_shutdown(smc->tcpsocket, SHUT_RDWR);
+			smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+			/* to wake up kernel_accept of smc_tcp_listen_worker */
+		}
+		release_sock(sk);
+		smc_sock_cleanup_listen(sk);
+		while (work_busy(&smc->tcp_listen_work) ||
+		       work_busy(&smc->listen_work)) {
+			msleep(20);
+		}
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+		break;
+	case SMC_ACTIVE:
+		/* active close */
+		if (conn->local_rx_ctrl.conn_state_flags.sending_done)
+			sk->sk_state = SMC_PEERCLW2;
+		else
+			sk->sk_state = SMC_PEERCLW1;
+		sk->sk_state_change(sk);
+		/* handle SOCK_LINGER */
+		/* fall through */
+	case SMC_PEERCLW1:
+		if (wait)
+			smc_wait_close(smc);
+		sk->sk_state = SMC_PEERCLW2;
+		smc_send_close(smc);
+		/* fall through */
+	case SMC_PEERCLW2:
+		rc = smc_wait_for_pending_sends(smc);
+		if ((rc <= 0) ||   /* timeout or interrupted */
+		    smc_close_received(conn)) {
+			sk->sk_state = SMC_CLOSED;
+			sk->sk_state_change(sk);
+		}
+		break;
+	case SMC_PROCESSABORT:
+	case SMC_APPLFINCLW:
+		/* socket already shutdown wr or both (active close) */
+		if (!conn->local_tx_ctrl.conn_state_flags.closed_conn &&
+		    conn->lgr) {
+			rc = smc_send_close(smc);
+			smc_wait_for_pending_sends(smc);
+		}
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_state_change(sk);
+		break;
+	case SMC_APPLCLW1:
+		/* passive close */
+		if (wait)
+			smc_wait_close(smc);
+		/* fall through */
+	case SMC_APPLCLW2:
+		/* confirm close from peer */
+		smc_send_close(smc);
+		if (smc_close_received(conn)) {
+			smc_wait_for_pending_sends(smc);
+			sk->sk_state = SMC_CLOSED;
+			sk->sk_state_change(sk);
+		} else {
+			sk->sk_state = SMC_PEERFINCLW;
+			sk->sk_state_change(sk);
+		}
+		break;
+	case SMC_PEERFINCLW:
+		break;
+	default:
+		break;
+	}
+out:
+	return;
+}
+
+/* determine vitality of peer
+ * returns 0 if peer is alive
+ */
+int smc_check_peer_gone(struct smc_link_group *lgr)
+{
+	struct smc_link *link;
+	struct llc_qentry *qentry = NULL;
+	int i, rc = 0;
+
+	if (!lgr)
+		return -EPIPE;
+	if (signal_pending(current))
+		return 0;
+	/* check peer with testlink */
+	if (!llc_initiate(lgr, LLC_GRP_TEST_LINK))
+		return -ETIMEDOUT;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		link = &lgr->lnk[i];
+		if (atomic_read(&link->state) != SMC_LINK_UP)
+			continue;
+		rc = llc_send_test_link(link, "KEEPALIVE_CHECK");
+		if (rc)
+			continue;
+		qentry = llc_wait(lgr, LLC_TESTLINK, LLC_WAIT_TIMER_TIMEO,
+				  LLC_TEST_LINK);
+		if (qentry) { /* peer is alive */
+			kfree(qentry);
+			goto out;
+		}
+	}
+	rc = -ETIMEDOUT;
+out:
+	llc_stop(lgr, LLC_GRP_TEST_LINK);
+	return rc;
+}
+
+static int smc_sock_release(struct socket *sock)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc;
+	int rc = 0;
+
+	if (!sk || (sk->sk_state == SMC_DESTRUCT))
+		return rc;
+	smc = smc_sk(sk);
+
+	sock_hold(sk);
+	if (sk->sk_state == SMC_LISTEN)
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+	else
+		lock_sock(sk);
+	smc_sk_release(smc, 1);
+	if (smc->tcpsocket) {
+		struct socket *tcp;
+
+		tcp = smc->tcpsocket;
+		smc->tcpsocket = NULL;
+		sock_release(tcp);
+	}
+
+	/* detach socket */
+	sock->state = SS_UNCONNECTED;
+	sock_set_flag(sk, SOCK_ZAPPED);
+	sock_orphan(sk); /* incl. SOCK_DEAD */
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+	}
+	release_sock(sk);
+	if (!smc->use_tcp && (smc->write_work.work.func == smc_write_worker)) {
+		wake_up_interruptible(&smc->rx_waiter);
+		cancel_delayed_work_sync(&smc->write_work);
+	}
+	if (sk->sk_state == SMC_PEERCLW2) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			/* guarantee socket cleanup */
+			mod_timer_pending(&smc->fin_timer,
+					  jiffies + TCP_FIN_TIMEOUT);
+			smc_terminate_conn(smc->conn.lgr);
+		}
+	}
+
+	/* check if we can cleanup sock or keep it for peers responses */
+	if ((sk->sk_state == SMC_CLOSED) &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		if (timer_pending(&smc->fin_timer))
+			mod_timer_pending(&smc->fin_timer, jiffies);
+		schedule_work(&smc->release_work);
+	}
+	sock->sk = NULL;
+	sock_put(sk);
+
+	return rc;
+}
+
+static int smc_sock_bind(struct socket *sock, struct sockaddr *addr,
+			 int addr_len)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc;
+
+	smc->tcpsocket->sk->sk_reuse = sk->sk_reuse;
+	if (addr->sa_family == AF_SMC)
+		addr->sa_family = AF_INET;
+	rc = kernel_bind(smc->tcpsocket, addr, addr_len);
+	return rc;
+}
+
+/* Wait for data on the tcp-socket and fallback to tcp in decline is received
+ * Returns 0 if success
+ *         tcpsocket error, -ETIMEDOUT, -EPROTO or -ECONNABORTED otherwise
+ */
+int clc_wait_msg(struct smc_sock *smc, int noblock, char *buf, int buflen,
+		 u8 type, int *reason_code)
+{
+	struct sock *sk = smc->tcpsocket->sk;
+	long timeo;
+	int rc = 0;
+	int len;
+	int krflags = 0;
+	struct msghdr msg;
+	struct kvec vec;
+	struct smc_clc_msg_hdr *clcm = (struct smc_clc_msg_hdr *)buf;
+	DEFINE_WAIT(wait);
+
+	timeo = (noblock) ? 0 : LLC_WAIT_FIRST_TIMEO + 1 * HZ;
+	if (noblock)
+		krflags = MSG_DONTWAIT;
+
+again:
+	lock_sock(sk);
+	rc = sk_wait_data(sk, &timeo);
+	release_sock(sk);
+	if (sk->sk_err)
+		rc = -sk->sk_err;
+	if (signal_pending(current)) {
+		rc = -EINTR;
+		sk->sk_err = -rc;
+	}
+	if (rc < 0) {
+		smc->sk.sk_err = sk->sk_err;
+		goto out;
+	}
+	if ((rc == 0) && !noblock) {
+		if (timeo) {
+			goto again;
+		} else {
+			smc->sk.sk_err = ETIMEDOUT;
+			rc = -ETIMEDOUT;
+			goto out;
+		}
+	}
+	rc = 0;
+	vec.iov_base = buf;
+	vec.iov_len = buflen;
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_recvmsg(smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_clc_msg_hdr), krflags);
+	if (!len) { /* no data */
+		smc->sk.sk_err = ETIMEDOUT;
+		rc = -ETIMEDOUT;
+		goto out;
+	}
+	if (len < sizeof(struct smc_clc_msg_hdr)) {
+		smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	len = ntohs(clcm->length);
+	if (!len) {
+		smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	vec.iov_base = buf + sizeof(struct smc_clc_msg_hdr);
+	vec.iov_len = buflen - sizeof(struct smc_clc_msg_hdr);
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_recvmsg(smc->tcpsocket, &msg, &vec, 1,
+			     len - sizeof(struct smc_clc_msg_hdr), 0);
+	if (len <= 0) {
+		smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	len = len + sizeof(struct smc_clc_msg_hdr);
+	if ((len < sizeof(struct smc_decline_clc_msg)) ||
+	    memcmp(clcm->ec_clce, SMC_EC, sizeof(SMC_EC))) {
+		if (!sk->sk_err)
+			sk->sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+	if (clcm->type == SMC_CLC_DECLINE) {
+		smc->use_tcp = 1;
+		if (clcm->flags & SMC_LINK_GROUP_OUT_OF_SYNCH) {
+			smc_terminate_conn(smc->conn.lgr);
+			sk->sk_err = ECONNABORTED;
+			rc = -ECONNABORTED;
+			return rc;
+		}
+		*reason_code = SMC_CLC_DEC_REPLY;
+		goto out;
+	}
+	if ((clcm->type != type) ||
+	    (memcmp(&clcm->ec_clce, SMC_EC, sizeof(SMC_EC)))) {
+		if (!smc->sk.sk_err)
+			smc->sk.sk_err = EPROTO;
+		rc = -EPROTO;
+		goto out;
+	}
+out:
+	return rc;
+}
+
+int clc_send_decline(struct smc_sock *smc, u32 reason_code, u8 flags)
+{
+	struct smc_decline_clc_msg dclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int len;
+
+	memset(&dclc, 0, sizeof(struct smc_decline_clc_msg));
+	memcpy(dclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(dclc.peer_id, local_peerid, sizeof(local_peerid));
+	dclc.hdr.type = SMC_CLC_DECLINE;
+	dclc.hdr.length = htons(sizeof(struct smc_decline_clc_msg));
+	dclc.hdr.flags = SMC_CLC_V1 | flags;
+	dclc.peer_diagnosis = htonl(reason_code);
+	memcpy(dclc.ec_dclcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &dclc;
+	vec.iov_len = sizeof(struct smc_decline_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_decline_clc_msg));
+	if (len < sizeof(struct smc_decline_clc_msg))
+		smc->sk.sk_err = EPROTO;
+	if (len < 0)
+		smc->sk.sk_err = -len;
+	return len;
+}
+
+/* Copy flags relevant for SO_* sockopts */
+static void smc_copy_sockopt_settings(struct sock *nsk, struct sock *osk)
+{
+	nsk->sk_type = osk->sk_type;
+	nsk->sk_sndbuf = osk->sk_sndbuf;
+	nsk->sk_rcvbuf = osk->sk_rcvbuf;
+	nsk->sk_sndtimeo = osk->sk_sndtimeo;
+	nsk->sk_rcvtimeo = osk->sk_rcvtimeo;
+	nsk->sk_mark = osk->sk_mark;
+	nsk->sk_priority = osk->sk_priority;
+	nsk->sk_rcvlowat = osk->sk_rcvlowat;
+	nsk->sk_bound_dev_if = osk->sk_bound_dev_if;
+	sock_copy_flags(nsk, osk);
+}
+
+static void smc_init_tmp_sockopts(struct smc_sock *smc)
+{
+	smc->tmp_sockopts.set = 0;
+	smc->tmp_sockopts.cork = 0;
+	smc->tmp_sockopts.defer_accept = 0;
+	smc->tmp_sockopts.nodelay = 0;
+}
+
+static int smc_sock_setsockopt(struct socket *, int, int, char __user *,
+			       unsigned int);
+
+/* Apply all user-set sockopts effective that were supressed as they
+ * might have a negative effect during the CLC handshake
+ */
+static int smc_apply_tmp_sockopts(struct socket *socket,
+				  const struct smc_tmp_sockopts *sopts)
+{
+	int rc = 0, brc;
+	mm_segment_t old_fs = get_fs();
+
+	set_fs(KERNEL_DS);
+	if (sopts->set & SMC_SOCKOPTS_CORK) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_CORK,
+					  (char __user *)&sopts->cork,
+					  sizeof(sopts->cork));
+		if (brc)
+			rc = -EFAULT;
+	}
+	if (sopts->set & SMC_SOCKOPTS_DEFER_ACCEPT) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_DEFER_ACCEPT,
+					  (char __user *)&sopts->defer_accept,
+					  sizeof(sopts->defer_accept));
+		if (brc)
+			rc = -EFAULT;
+	}
+	if (sopts->set & SMC_SOCKOPTS_NODELAY) {
+		brc = smc_sock_setsockopt(socket, SOL_TCP, TCP_NODELAY,
+					  (char __user *)&sopts->nodelay,
+					  sizeof(sopts->nodelay));
+		if (brc)
+			rc = -EFAULT;
+	} else {
+		struct smc_sock *smc = smc_sk(socket->sk);
+
+		tcp_sk(smc->tcpsocket->sk)->nonagle &= ~TCP_NAGLE_OFF;
+	}
+	set_fs(old_fs);
+
+	return rc;
+}
+
+static int smc_find_link_accept(struct smc_link_group *lgr,
+				struct smc_acc_conf_clc_msg *clc)
+{
+	int i;
+	u32 tmp_qpnum;
+
+	ntoh_three(&tmp_qpnum, clc->qp_num);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (!memcmp(&lgr->lnk[i].gid_peer, &clc->hdr2.gid, 16) &&
+		    !memcmp(&lgr->lnk[i].mac_peer, &clc->hdr2.mac, 6) &&
+		    (lgr->lnk[i].qp_num_peer == tmp_qpnum))
+			return i;
+	}
+	return -ENOENT;
+}
+
+static int smc_rtoken_handling(struct smc_link_group *lgr, struct smc_sock *smc,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	struct smc_connection *conn = &smc->conn;
+	u64 vaddr = be64_to_cpu(clc->rmb_vaddr);
+	unsigned int rkey = ntohl(clc->rmb_rkey);
+	int lnk_idx;
+
+	lnk_idx = smc_find_link_accept(lgr, clc);
+	if (lnk_idx < 0)
+		return -ENOENT;
+	conn->rtok_idx =
+		smc_find_rtoken_by_link(lgr, lnk_idx, rkey);
+	if (conn->rtok_idx == -ENOENT) {
+		conn->rtok_idx = smc_get_rtoken(lgr);
+		lgr->rtok[conn->rtok_idx].rkey[lnk_idx] = rkey;
+		lgr->rtok[conn->rtok_idx].vaddr[lnk_idx] = vaddr;
+		lgr->rtok[conn->rtok_idx].link_id[lnk_idx] =
+			lgr->lnk[lnk_idx].link_id;
+	}
+	return 0;
+}
+
+static int smc_send_proposal(struct smc_sock *smc,
+			     struct smc_roce_defs *roce_sugg,
+			     struct smc_acc_conf_clc_msg *aclc,
+			     int *reason_code)
+{
+	int pclc_msgsize = sizeof(struct smc_proposal_clc_msg) +
+			   sizeof(struct smc_proposal_clc_msg2);
+	struct smc_proposal_clc_msg *pclc;
+	struct smc_proposal_clc_msg2 *pclc2;
+	struct kvec vec;
+	struct msghdr msg;
+	int rc = 0;
+
+	pclc = kzalloc(pclc_msgsize, GFP_KERNEL);
+	if (!pclc)
+		return -ENOMEM;
+	/* send SMC Proposal CLC message (peer ID, GID, MAC, VLAN) */
+	memcpy(&pclc->hdr2.gid, &roce_sugg->gid, sizeof(roce_sugg->gid));
+	memcpy(&pclc->hdr2.mac, &roce_sugg->mac, sizeof(roce_sugg->mac));
+	pclc->iparea_offset = 0;
+
+	pclc2 = (struct smc_proposal_clc_msg2 *)
+		((char *)pclc + sizeof(struct smc_proposal_clc_msg) +
+		 pclc->iparea_offset);
+
+	/* is it sufficient to use IP from connection itself? */
+	rc = smc_netinfo_by_tcpsk(&pclc2->outgoing_subnet,
+				  &pclc2->subnet_mask_bits, smc->tcpsocket);
+	memcpy(&pclc->hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(&pclc->hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	pclc->hdr.type = SMC_CLC_PROPOSAL;
+	pclc->hdr.length = htons(pclc_msgsize);
+	pclc->hdr.flags = SMC_CLC_V1;
+	memcpy(&pclc2->ec_pclcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = pclc;
+	vec.iov_len = pclc_msgsize;
+	memset(&msg, 0, sizeof(struct msghdr));
+	rc = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1, pclc_msgsize);
+	kfree(pclc);
+	if (rc >= 0) {
+		if (rc < pclc_msgsize)
+			rc = -EPROTO;
+		else
+			rc = 0;
+	}
+	if (rc) {
+		smc->sk.sk_err = -rc;
+		return rc;
+	}
+
+	/* receive SMC Accept CLC message (GID, MAC, VLAN, QP )*/
+	rc = clc_wait_msg(smc, SMC_CLC_BLOCK, (char *)aclc,
+			  sizeof(struct smc_acc_conf_clc_msg),
+			  SMC_CLC_ACCEPTANCE, reason_code);
+	return rc;
+}
+
+static int smc_send_confirm(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct smc_connection *conn = &smc->conn;
+	struct smc_acc_conf_clc_msg cclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int rc = 0;
+
+	link = &conn->lgr->lnk[conn->lnk_idx];
+	lgr = link->lnk_grp;
+	/* send SMC Confirm CLC msg (MAC, GID, QP, link num, link user id) */
+	memset(&cclc, 0, sizeof(struct smc_acc_conf_clc_msg));
+	cclc.tcp_conn_idx = 1; /* as long as 1 RMB = 1 RMBE */
+	memcpy(cclc.hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	memcpy(&cclc.hdr2.gid, &link->roce_defs.gid,
+	       sizeof(link->roce_defs.gid));
+	memcpy(&cclc.hdr2.mac, &link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+	hton_three(link->roce_qp->qp_num, cclc.qp_num);
+	hton_three(link->psn_initial, cclc.psn);
+	cclc.rmb_rkey =
+		htonl(conn->rmb_rx_elem->mr_rx[conn->lnk_idx]->rkey);
+	cclc.rmb_vaddr =
+		cpu_to_be64((u64)conn->rmb_rx_elem->rmb_rx_dma[conn->lnk_idx]);
+	memcpy(cclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	cclc.hdr.type = SMC_CLC_CONFIRMATION;
+	cclc.hdr.length = htons(sizeof(struct smc_acc_conf_clc_msg));
+	cclc.hdr.flags = SMC_CLC_V1;
+	cclc.rmb_alert_token = conn->alert_token_local;
+	cclc.flags2 = conn->rmb_rx_size_short +
+		      min(link->path_mtu, link->mtu_peer);
+	memcpy(cclc.ec_clcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &cclc;
+	vec.iov_len = sizeof(struct smc_acc_conf_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	rc = kernel_sendmsg(smc->tcpsocket, &msg, &vec, 1,
+			    sizeof(struct smc_acc_conf_clc_msg));
+	if (rc >= 0) {
+		if (rc < sizeof(struct smc_acc_conf_clc_msg))
+			rc = -EPROTO;
+		else
+			rc = 0;
+	}
+	if (rc)
+		smc->sk.sk_err = -rc;
+	return rc;
+}
+
+static int smc_initial_map_rmb(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &lgr->lnk[smc->conn.lnk_idx];
+	long rc = 0;
+
+	if (!smc->conn.tx_rmb_reused) {
+		rc = smc_map_rmbs_tx(lgr, smc->conn.lnk_idx,
+				     smc->conn.rmb_tx_size,
+				     smc->conn.rmb_tx_elem);
+		if (rc)
+			goto out;
+	}
+	if (!smc->conn.rx_rmb_reused) {
+		rc = smc_map_rmbs_rx(lgr, smc->conn.lnk_idx,
+				     smc->conn.rmb_rx_size,
+				     smc->conn.rmb_rx_elem);
+		if (rc)
+			goto out;
+		rc = smc_get_dma_mr(lgr, smc->conn.lnk_idx,
+				    smc->conn.rmb_rx_elem);
+		if (rc)
+			goto out;
+	}
+	return 0;
+
+out:
+	if (rc && smc_lnk_downing(&link->state)) {
+		release_sock(&smc->sk);
+		smc_link_down(lgr, link - &lgr->lnk[0]);
+		lock_sock(&smc->sk);
+	}
+	return (int)rc;
+}
+
+static int smc_announce_rmb(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* Announce new RMB */
+	if (smc->conn.tx_rmb_reused && smc->conn.rx_rmb_reused)
+		goto out;
+	if (!llc_initiate(lgr, LLC_GRP_CONF_RKEY)) { /* timeout */
+		*reason_code = SMC_CLC_DEC_TU;
+		rc = -ETIMEDOUT;
+		goto out;
+	}
+	smc_map_rmbs(smc);
+	if (lgr->lgr_type == SINGLE)
+		goto out_stop;
+	if (smc->conn.rx_rmb_reused)
+		goto out_stop;
+	rc = llc_do_confirm_rkey(link, smc->conn.rmb_rx_elem);
+	if (rc >= 0) {
+		if (lgr->role == SMC_CLNT)
+			release_sock(&smc->sk);
+		qentry = llc_wait(lgr, lgr->role, LLC_WAIT_TIMEO,
+				  LLC_CONFIRM_RKEY);
+		if (lgr->role == SMC_CLNT)
+			lock_sock(&smc->sk);
+		if ((!qentry) ||
+		    (qentry->msg.hd.flags & LLC_FLAG_RKEY_NEG)) {
+			kfree(qentry);
+			if (smc_lnk_downing(&link->state)) {
+				if (lgr->role == SMC_CLNT)
+					release_sock(&smc->sk);
+				smc_link_down(lgr, link - &lgr->lnk[0]);
+				if (lgr->role == SMC_CLNT)
+					lock_sock(&smc->sk);
+			}
+			*reason_code = SMC_CLC_DEC_TU;
+			rc = -ETIMEDOUT;
+			goto out_stop;
+		}
+		kfree(qentry);
+		rc = 0;
+	} else {
+		*reason_code = SMC_CLC_DEC_SEND;
+		rc = -EPIPE;
+	}
+out_stop:
+	llc_stop(lgr, LLC_GRP_CONF_RKEY);
+out:
+	if (rc)
+		smc_remove_rx_rmb(smc);
+	return rc;
+}
+
+static int smc_clnt_conf_first_lnk(struct smc_sock *smc,
+				   struct smc_roce_defs *roce_sugg,
+				   int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_confirm_msg *conf_llc;
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* receive CONFIRM LINK request over RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_CONFIRM_LINK);
+	if (!qentry) {
+		/* clc decl received or llc timeout */
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_NO_NOTIFY, smc);
+		return rc;
+	}
+	if (!(qentry->msg.hd.flags & 0xf)) {
+		/* no cache alignment specified, decline */
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		*reason_code = SMC_CLC_DEC_INT;
+		rc = -EOPNOTSUPP;
+		kfree(qentry);
+		return rc;
+	}
+	lgr->cache_align_size =
+		0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+	conf_llc = (struct llc_confirm_msg *)&qentry->msg;
+	link->link_id = conf_llc->link_num;
+	lgr->rtok[smc->conn.rtok_idx].link_id[smc->conn.lnk_idx] =
+		link->link_id;
+	lgr->max_links = SMC_MAX_SYM_LINKS;
+	kfree(qentry);
+	rc = smc_modify_qp_rts(link);
+	if (rc) {
+		*reason_code = SMC_CLC_DEC_INT;
+		return rc;
+	}
+	smc_check_qp_attr(link);
+	/* send CONFIRM LINK response over RoCE fabric */
+	rc = llc_send_confirm_link(link, (u8 *)link->roce_defs.mac,
+				   &roce_sugg->gid, LLC_RESP, LLC_SERV);
+	if (rc < 0) {
+		*reason_code = SMC_CLC_DEC_TCL;
+		return rc;
+	}
+	atomic_set(&link->state, SMC_LINK_UP);
+	return rc;
+}
+
+static int smc_clnt_add_lnk(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	release_sock(&smc->sk);
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_ADD_LINK);
+	lock_sock(&smc->sk);
+	if (!qentry) {
+		lgr->lgr_type = NONE;
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_SERV, smc);
+		return rc;
+	}
+	rc = llc_cli_add_link(link, qentry, smc);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+	if (rc == -ENOLINK)
+		*reason_code = SMC_LLC_DEL_NOLNK;
+	return rc;
+}
+
+static void smc_save_peer_info(struct smc_sock *smc,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	smc->conn.local_tx_ctrl.ctrl_token = clc->rmb_alert_token;
+	smc->conn.peer_rx_buf_len =
+		smc_uncompress_bufsize(clc->flags2 & 0xf0);
+	smc->conn.peer_tcp_conn_idx = clc->tcp_conn_idx;
+	atomic_set(&smc->conn.rx_buf_space, smc->conn.peer_rx_buf_len);
+}
+
+static void smc_save_link_info(struct smc_link *link,
+			       struct smc_acc_conf_clc_msg *clc)
+{
+	memcpy(link->gid_peer, &clc->hdr2.gid, sizeof(link->gid_peer));
+	memcpy(link->mac_peer, clc->hdr2.mac, sizeof(link->mac_peer));
+	ntoh_three(&link->psn_peer, clc->psn);
+	link->mtu_peer = clc->flags2 & 0x0f;
+}
+
+static int smc_switch_to_tcp(struct smc_sock *smc, int reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	int rc;
+
+	smc->use_tcp = 1;
+	if (reason_code && (reason_code != SMC_CLC_DEC_REPLY)) {
+		rc = clc_send_decline(smc, reason_code, 0);
+		if (rc < sizeof(struct smc_decline_clc_msg))
+			return rc;
+	}
+	if (lgr) {
+		if (smc->new_lgr && link) {
+			atomic_set(&link->state, SMC_LINK_DOWN);
+			lgr->lgr_type = NONE;
+		}
+		if (lgr->lgr_type == NONE) {
+			spin_lock_bh(&smc_lgr_list.lock);
+			list_del_init(&lgr->list);
+			spin_unlock_bh(&smc_lgr_list.lock);
+		}
+	}
+	smc_free_conn(smc);
+	return 0;
+}
+
+static void smc_init_after_clc_handshake(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	u32 keepalive_time;
+
+	init_waitqueue_head(&smc->prep_waiter);
+	init_waitqueue_head(&smc->destruct_waiter);
+	if (sk->sk_state == SMC_INIT)
+		sk->sk_state = SMC_ACTIVE;
+	if (smc->use_tcp)
+		goto out;
+	INIT_DELAYED_WORK(&smc->write_work, smc_write_worker);
+	if (sock_flag(sk, SOCK_KEEPOPEN)) {
+		keepalive_time = tp->keepalive_time ? : TCP_KEEPALIVE_TIME;
+		if (keepalive_time)
+			sk_reset_timer(sk, &sk->sk_timer,
+				       jiffies + (unsigned long)keepalive_time);
+	}
+	sk->sk_data_ready = smc_sock_wake_rx;
+	sk->sk_write_space = smc_sock_wake_tx;
+	/* to return meaningful values with getsockopt */
+	sk->sk_sndbuf = smc->conn.rmb_tx_size;
+	sk->sk_rcvbuf = smc->conn.rmb_rx_size;
+out:
+	return;
+}
+
+static int smc_connect(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct socket *sock = sk->sk_socket;
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	int rc = 0;
+	struct sockaddr_in *inaddr =  (struct sockaddr_in *)smc->addr;
+	struct smc_acc_conf_clc_msg aclc;
+	struct smc_link_group *lgr = NULL;
+	struct smc_link *link = NULL;
+	struct smc_roce_defs roce_sugg;
+	u32 tmp_qpnum;
+	char pnet_id[SMC_MAX_PNET_ID_LEN];
+	int reason_code = 0;
+
+	/* check if peer is smc capable */
+	if (!tp->syn_smc) {
+		smc->use_tcp = 1;
+		goto out_connected;
+	}
+
+	smc_find_roce_resources(&roce_sugg, pnet_id, inaddr,
+				smc->tcpsocket->sk);
+	if (!roce_sugg.ibdev) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+	/* do inband token exchange */
+	rc = smc_send_proposal(smc, &roce_sugg, &aclc, &reason_code);
+	if (rc) {
+		if ((rc == -EPROTO) || (rc == -ETIMEDOUT)) {
+			smc->sk.sk_err = ENETUNREACH;
+			rc = -ENETUNREACH;
+		}
+		goto out_err;
+	}
+	if (reason_code) { /* switch to tcp */
+		smc->use_tcp = 1;
+		goto out_connected;
+	}
+
+	/* allocate connection/link group*/
+	ntoh_three(&tmp_qpnum, aclc.qp_num);
+	smc->new_lgr = aclc.hdr.flags & SMC_FIRST_CONTACT;
+	rc = smc_create_conn(smc, inaddr->sin_addr.s_addr, SMC_CLNT,
+			     &roce_sugg, &aclc.hdr2, tmp_qpnum);
+	if (rc == -ENOLINK) { /* link groups out of sync */
+		smc->sk.sk_err = ENETUNREACH;
+		rc = -ENETUNREACH;
+		goto out_err;
+	}
+	if (rc) {
+		/* insufficient memory */
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	lgr = smc->conn.lgr;
+	link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	strncpy(lgr->pnet_id, pnet_id, sizeof(pnet_id));
+	smc_save_peer_info(smc, &aclc);
+	lock_sock(sk);
+
+	rc = smc_create_rmbs(smc);
+	if (rc) {
+		/* insufficient memory */
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	atomic_set(&smc->conn.tx_buf_space, smc->conn.rmb_tx_size);
+	if (smc->new_lgr) {
+		rc = smc_initial_map_rmb(smc);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_RIM;
+			goto decline;
+		}
+		link->qp_num_peer = tmp_qpnum;
+		smc_save_link_info(link, &aclc);
+	}
+	rc = smc_rtoken_handling(lgr, smc, &aclc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_INT;
+		goto decline;
+	}
+	if (smc->new_lgr) {
+		rc = smc_ready_link(link);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_INT;
+			goto decline;
+		}
+		llc_initiate(lgr, LLC_GRP_ADD_LINK);
+		lgr->lgr_type = SINGLE;
+	} else { /* link already active */
+		rc = smc_announce_rmb(smc, &reason_code);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_TU;
+			goto decline;
+		}
+	}
+	rc = smc_send_confirm(smc);
+	if (rc)
+		goto out_err;
+
+	if (smc->new_lgr) {
+		/* QP confirmation over RoCE fabric */
+		rc = smc_clnt_conf_first_lnk(smc, &roce_sugg, &reason_code);
+		if (rc == -EINTR)
+			goto out_err;
+		if (rc)
+			goto decline;
+		/* receive ADD LINK request over RoCE fabric */
+		rc = smc_clnt_add_lnk(smc, &reason_code);
+		if (rc == -ENOLINK)
+			goto out_connected;
+		if (rc == -ETIMEDOUT)
+			goto out_err;
+		if (rc)
+			goto decline;
+	}
+
+out_connected:
+	sock->state = SS_CONNECTED;
+	smc_init_after_clc_handshake(smc);
+	/* copy setsockopt settings to tcpsocket to be prepared for fallback */
+	smc_copy_sockopt_settings(smc->tcpsocket->sk, sk);
+	/* to return meaningful values with getsockopt */
+	if (lgr) {
+		release_sock(sk);
+		if (mutex_is_locked(&lgr->conf_mutex))
+			mutex_unlock(&lgr->conf_mutex);
+	}
+	smc_apply_tmp_sockopts(sk->sk_socket, &smc->tmp_sockopts);
+	return 0;
+
+decline:
+	rc = smc_switch_to_tcp(smc, reason_code);
+	if (!rc)
+		goto out_connected;
+out_err:
+	if (lgr) {
+		if (smc->new_lgr)
+			lgr->lgr_type = NONE;
+		release_sock(sk);
+		if (mutex_is_locked(&lgr->conf_mutex))
+			mutex_unlock(&lgr->conf_mutex);
+		smc_free_conn(smc);
+	}
+	smc->use_tcp = 1;
+	sk->sk_state = SMC_CLOSED;
+	return rc;
+}
+
+static int smc_sock_connect(struct socket *sock, struct sockaddr *addr,
+			    int alen, int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	int rc, loop = 0;
+
+	/* set SMC capability on tcp-socket */
+	tp->syn_smc = 1;
+	smc->addr = addr;
+	if (addr->sa_family == AF_SMC)
+		addr->sa_family = AF_INET;
+
+again:
+	rc = kernel_connect(smc->tcpsocket, addr, alen, flags);
+	if (rc == -ERESTARTSYS) {
+		msleep(1000);
+		loop++;
+		if (loop < 10)
+			goto again;
+		rc = -EAGAIN;
+	}
+	if (rc == -EINPROGRESS)
+		return rc;
+
+	if (!rc) {
+		rc = smc_connect(smc);
+		if (rc)
+			rc = -1;
+	} else {
+		smc->use_tcp = 1;
+		sk->sk_state = SMC_CLOSED;
+		sk->sk_err = -rc;
+		sk->sk_state_change(sk);
+		rc = -1;
+	}
+
+	return rc;
+}
+
+static int smc_sock_listen(struct socket *sock, int backlog)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	int rc;
+
+	lock_sock(sk);
+	/* copy setsockopt settings to tcpsocket */
+	smc_copy_sockopt_settings(smc->tcpsocket->sk, sk);
+	/* set SMC capability on tcp-socket */
+	tp->syn_smc = 1;
+	rc = kernel_listen(smc->tcpsocket, backlog);
+
+	if (!rc) {
+		sk->sk_max_ack_backlog = backlog;
+		sk->sk_ack_backlog = 0;
+		sk->sk_state = SMC_LISTEN;
+		if (!smc->use_tcp) {
+			INIT_WORK(&smc->tcp_listen_work, smc_tcp_listen_worker);
+			schedule_work(&smc->tcp_listen_work);
+		}
+	}
+
+	release_sock(sk);
+	return rc;
+}
+
+static void smc_accept_enqueue(struct sock *parent, struct sock *sk)
+{
+	struct smc_sock *par = smc_sk(parent);
+
+	sock_hold(sk);
+	spin_lock(&par->accept_q_lock);
+	list_add_tail(&smc_sk(sk)->accept_q, &par->accept_q);
+	spin_unlock(&par->accept_q_lock);
+	sk_acceptq_added(parent);
+}
+
+static void smc_accept_unlink(struct sock *sk)
+{
+	struct smc_sock *par = smc_sk(sk)->listen_smc;
+
+	spin_lock(&par->accept_q_lock);
+	list_del_init(&smc_sk(sk)->accept_q);
+	spin_unlock(&par->accept_q_lock);
+	sk_acceptq_removed(&smc_sk(sk)->listen_smc->sk);
+	sock_put(sk);
+}
+
+static struct sock *smc_accept_dequeue(struct sock *parent,
+				       struct socket *newsock)
+{
+	struct smc_sock *isk, *n;
+	struct sock *sk;
+
+	list_for_each_entry_safe(isk, n, &smc_sk(parent)->accept_q, accept_q) {
+		sk = (struct sock *)isk;
+		lock_sock(sk);
+
+		smc_accept_unlink(sk);
+		if (newsock)
+			sock_graft(sk, newsock);
+		release_sock(sk);
+		return sk;
+	}
+	return NULL;
+}
+
+static int smc_sock_accept(struct socket *sock, struct socket *newsock,
+			   int flags)
+{
+	DECLARE_WAITQUEUE(wait, current);
+	struct sock *sk = sock->sk, *nsk;
+	long timeo;
+	int rc = 0;
+
+	lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+	timeo = sock_rcvtimeo(sk, flags & O_NONBLOCK);
+
+	/* Wait for an incoming connection */
+	add_wait_queue_exclusive(sk_sleep(sk), &wait);
+	while (!(nsk = smc_accept_dequeue(sk, newsock))) {
+		set_current_state(TASK_INTERRUPTIBLE);
+		if (!timeo) {
+			rc = -EAGAIN;
+			break;
+		}
+
+		release_sock(sk);
+		timeo = schedule_timeout(timeo);
+		lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+
+		if (signal_pending(current)) {
+			rc = -EINTR;
+			break;
+		}
+	}
+
+	set_current_state(TASK_RUNNING);
+	remove_wait_queue(sk_sleep(sk), &wait);
+
+	if (rc) {
+		release_sock(sk);
+		goto done;
+	}
+	rc = sock_error(nsk);
+	if (rc) {
+		release_sock(sk);
+		goto done;
+	}
+
+	newsock->state = SS_CONNECTED;
+	release_sock(sk);
+	smc_apply_tmp_sockopts(newsock, &smc_sk(sk)->tmp_sockopts);
+
+	/* don't get value from tmp_sockopts as TCP will mangle the value */
+	timeo = 0;
+	if (smc_sk(sk)->tmp_sockopts.set & SMC_SOCKOPTS_DEFER_ACCEPT)
+		timeo = smc_getsockopt_from_tcp(smc_sk(nsk), TCP_DEFER_ACCEPT);
+
+	if (!rc && timeo && !(flags & O_NONBLOCK)) {
+		int rc = 0;
+
+		timeo = msecs_to_jiffies(timeo * 1000);
+		if (smc_sk(nsk)->use_tcp) {
+			lock_sock(smc_sk(nsk)->tcpsocket->sk);
+			rc = sk_wait_data(smc_sk(nsk)->tcpsocket->sk, &timeo);
+			release_sock(smc_sk(nsk)->tcpsocket->sk);
+		} else {
+			lock_sock(nsk);
+			rc = smc_wait_rx_data(smc_sk(nsk), 1, 1, timeo);
+			release_sock(nsk);
+		}
+	}
+done:
+	return rc;
+}
+
+static int smc_getsockopt_from_tcp(struct smc_sock *smc, int optname)
+{
+	char optval[10];
+	int optlen;
+	int rc;
+
+	optlen = sizeof(optlen);
+	memset(optval, 0, sizeof(optval));
+	rc = kernel_getsockopt(smc->tcpsocket, SOL_TCP, optname, optval,
+			       &optlen);
+	if (rc)
+		return -EINVAL;
+	memcpy(&rc, optval, optlen);
+	return rc;
+}
+
+static void smc_keepalive_timer(unsigned long);
+static void smc_sock_destruct(struct sock *);
+
+static int smc_send_accept(struct smc_sock *new_smc,
+			   struct smc_acc_conf_clc_msg *cclc, int *reason_code)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct smc_connection *conn = &new_smc->conn;
+	struct smc_acc_conf_clc_msg aclc;
+	struct msghdr msg;
+	struct kvec vec;
+	int len, rc = 0;
+
+	link = &conn->lgr->lnk[conn->lnk_idx];
+	lgr = link->lnk_grp;
+	/* send SMC Accept CLC message (GID, MAC, VLAN, QP) */
+	memset(&aclc, 0, sizeof(struct smc_acc_conf_clc_msg));
+	aclc.hdr.flags = SMC_CLC_V1; /* smc version */
+	if (new_smc->new_lgr)
+		aclc.hdr.flags |= SMC_FIRST_CONTACT;
+	aclc.tcp_conn_idx = 1; /* as long as 1 RMB = 1 RMBE */
+	memcpy(&aclc.hdr2.gid, &link->roce_defs.gid,
+	       sizeof(link->roce_defs.gid));
+	memcpy(&aclc.hdr2.mac, link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+	hton_three(link->roce_qp->qp_num, aclc.qp_num);
+	hton_three(link->psn_initial, aclc.psn);
+	aclc.rmb_rkey =
+		htonl(conn->rmb_rx_elem->mr_rx[conn->lnk_idx]->rkey);
+	aclc.rmb_vaddr =
+		cpu_to_be64((u64)conn->rmb_rx_elem->rmb_rx_dma[conn->lnk_idx]);
+	memcpy(aclc.hdr.ec_clce, SMC_EC, sizeof(SMC_EC));
+	memcpy(aclc.hdr2.peer_id, local_peerid, sizeof(local_peerid));
+	aclc.hdr.type = SMC_CLC_ACCEPTANCE;
+	aclc.hdr.length = htons(sizeof(struct smc_acc_conf_clc_msg));
+	aclc.rmb_alert_token = conn->alert_token_local;
+	aclc.flags2 = conn->rmb_rx_size_short + link->path_mtu;
+	memcpy(aclc.ec_clcf, SMC_EC, sizeof(SMC_EC));
+
+	vec.iov_base = &aclc;
+	vec.iov_len = sizeof(struct smc_acc_conf_clc_msg);
+	memset(&msg, 0, sizeof(struct msghdr));
+	len = kernel_sendmsg(new_smc->tcpsocket, &msg, &vec, 1,
+			     sizeof(struct smc_acc_conf_clc_msg));
+	if (len < sizeof(struct smc_acc_conf_clc_msg)) {
+		*reason_code = SMC_CLC_DEC_SEND;
+		goto out;
+	}
+
+	/* receive SMC Confirm CLC message (MAC, GID, QP, link #, link user id)
+	 */
+	rc = clc_wait_msg(new_smc, SMC_CLC_BLOCK, (char *)cclc,
+			  sizeof(struct smc_acc_conf_clc_msg),
+			  SMC_CLC_CONFIRMATION, reason_code);
+out:
+	return rc;
+}
+
+static int smc_serv_conf_first_lnk(struct smc_sock *smc,
+				   struct smc_roce_defs *roce_sugg,
+				   int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct llc_qentry *qentry;
+	int rc = 0;
+
+	/* send CONFIRM LINK request over the RoCE fabric */
+	llc_initiate(lgr, LLC_GRP_ADD_LINK);
+	rc = llc_send_confirm_link(link, (u8 *)link->roce_defs.mac,
+				   &roce_sugg->gid, LLC_REQ, LLC_SERV);
+	if (rc < 0) {
+		*reason_code = SMC_CLC_DEC_TCL;
+		lgr->lgr_type = NONE;
+		goto out;
+	}
+	/* receive CONFIRM LINK response over the RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+			  LLC_CONFIRM_LINK);
+	if (!qentry) {
+		llc_stop(lgr, LLC_GRP_ADD_LINK);
+		rc = llc_get_fail_cause(lgr, LLC_SERV, smc);
+		atomic_set(&link->state, SMC_LINK_DOWN);
+		lgr->lgr_type = NONE;
+		if (rc == -ECOMM)
+			*reason_code = SMC_CLC_DEC_SEND;
+		if (!rc) /* clc decline received */
+			rc = -ECANCELED;
+	} else {
+		atomic_set(&link->state, SMC_LINK_UP);
+		if (!(qentry->msg.hd.flags & 0xf)) {
+			/* no cache alignment specified, decline */
+			kfree(qentry);
+			*reason_code = SMC_CLC_DEC_REPLY; /* no clc decline */
+			llc_send_del_link(link, link, LLC_FLAG_DEL_ORDERLY,
+					  LLC_DEL_PROT_VIOL, LLC_NO_NOTIFY);
+			atomic_set(&link->state, SMC_LINK_DOWN);
+			lgr->lgr_type = NONE;
+			schedule_delayed_work(&lgr->free_lgr_work,
+					      SMC_FREE_LGR_TIMEO);
+			rc = -EOPNOTSUPP;
+		} else {
+			lgr->cache_align_size = 0x00000001 <<
+				(((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+			kfree(qentry);
+		}
+	}
+out:
+	return rc;
+}
+
+static int smc_serv_add_lnk(struct smc_sock *smc, int *reason_code)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	struct smc_link *link = &smc->conn.lgr->lnk[smc->conn.lnk_idx];
+	struct smc_roce_defs roce_sugg;
+	int rc = 0;
+
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	roce_sugg.ibdev = NULL;
+	roce_sugg.port = 0;
+	roce_sugg.vlan = lgr->vlan;
+	smc_find_alt_roce_resources(&roce_sugg, lgr, link);
+	rc = llc_srv_add_link(link, &roce_sugg, smc);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+	return rc;
+}
+
+static void smc_listen_worker(struct work_struct *work)
+{
+	struct smc_sock *new_smc = container_of(work, struct smc_sock,
+						listen_work);
+	struct sock *newsmcsk = &new_smc->sk;
+	struct socket *newtcpsocket = new_smc->tcpsocket;
+	struct tcp_sock *ntp = tcp_sk(newtcpsocket->sk);
+	struct smc_sock *lsmc = new_smc->listen_smc;
+	struct sockaddr_in peeraddr;
+	char pclc_buf[sizeof(struct smc_proposal_clc_msg)
+		    + sizeof(struct smc_proposal_clc_msg2) + 40];
+	struct smc_proposal_clc_msg *pclc =
+		(struct smc_proposal_clc_msg *)&pclc_buf;
+	struct smc_proposal_clc_msg2 *pclc2;
+	struct smc_acc_conf_clc_msg cclc;
+	struct smc_link_group *lgr = NULL;
+	struct smc_link *link = NULL;
+	struct smc_roce_defs roce_sugg;
+	__be32 subnet = 0;
+	int reason_code = 0;
+	int rc = 0, len;
+	char pnet_id[SMC_MAX_PNET_ID_LEN];
+	u8 mlen = 0;
+
+	/* check if peer is smc capable */
+	if (!ntp->syn_smc) {
+		new_smc->use_tcp = 1;
+		goto out_connected;
+	}
+
+	kernel_getpeername(newtcpsocket, (struct sockaddr *)&peeraddr, &len);
+	new_smc->new_lgr = 0;
+	/* do inband token exchange
+	 * receive SMC Proposal CLC message (peer ID, GID, MAC, VLAN)
+	 */
+	rc = clc_wait_msg(new_smc, SMC_CLC_BLOCK, (char *)&pclc_buf,
+			  sizeof(pclc_buf), SMC_CLC_PROPOSAL, &reason_code);
+	if (rc)
+		goto out_err;
+	if (reason_code)
+			goto decline;
+
+	smc_find_roce_resources(&roce_sugg, pnet_id, &peeraddr,
+				newtcpsocket->sk);
+	if (!roce_sugg.ibdev) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+
+	/* is it sufficient to use IP from connection itself? */
+	rc = smc_netinfo_by_tcpsk(&subnet, &mlen, newtcpsocket);
+	pclc2 = (struct smc_proposal_clc_msg2 *)
+		((char *)(pclc) + sizeof(struct smc_proposal_clc_msg)
+	       + pclc->iparea_offset);
+	if ((pclc2->outgoing_subnet != subnet) ||
+	    (pclc2->subnet_mask_bits != mlen)) {
+		reason_code = SMC_CLC_DEC_CUC;
+		goto decline;
+	}
+
+	/* allocate connection/link group*/
+	rc = smc_create_conn(new_smc, peeraddr.sin_addr.s_addr,
+			     SMC_SERV, &roce_sugg, &pclc->hdr2, 0);
+	if (rc == -ENOLINK)
+		goto out_err;
+	if (rc) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	lgr = new_smc->conn.lgr;
+	link = &new_smc->conn.lgr->lnk[new_smc->conn.lnk_idx];
+
+	if (lgr->lgr_type == NONE) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	rc = smc_create_rmbs(new_smc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_RIM;
+		goto decline;
+	}
+	atomic_set(&new_smc->conn.tx_buf_space, new_smc->conn.rmb_tx_size);
+
+	/* Announce RMB Rtoken */
+	if (new_smc->new_lgr) {
+		rc = smc_initial_map_rmb(new_smc);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_RIM;
+			goto decline;
+		}
+	} else {
+		rc = smc_announce_rmb(new_smc, &reason_code);
+		if (rc)
+			goto decline;
+	}
+	strncpy(lgr->pnet_id, pnet_id, sizeof(pnet_id));
+	rc = smc_send_accept(new_smc, &cclc, &reason_code);
+	if (rc)
+		goto out_err;
+	if (reason_code)
+		goto decline;
+
+	smc_save_peer_info(new_smc, &cclc);
+	if (new_smc->new_lgr) {
+		ntoh_three(&link->qp_num_peer, cclc.qp_num);
+		smc_save_link_info(link, &cclc);
+	}
+	rc = smc_rtoken_handling(lgr, new_smc,
+				 (struct smc_acc_conf_clc_msg *)&cclc);
+	if (rc) {
+		reason_code = SMC_CLC_DEC_INT;
+		goto decline;
+	}
+
+	if (new_smc->new_lgr) {
+		rc = smc_ready_link(link);
+		if (rc) {
+			reason_code = SMC_CLC_DEC_INT;
+			goto decline;
+		}
+		lgr->max_links = SMC_MAX_SYM_LINKS;
+		lgr->lgr_type = SINGLE;
+		rc = smc_serv_conf_first_lnk(new_smc, &roce_sugg, &reason_code);
+		if ((rc == -EOPNOTSUPP) || (rc == -EINTR))
+			goto out_err; /* peer is not aware of a problem */
+		if (rc)
+			goto decline;
+		rc = smc_serv_add_lnk(new_smc, &reason_code);
+		if (rc == -ENOLINK)
+			goto out_connected;
+		else if (rc < 0)
+			goto out_err;
+	}
+
+out_connected:
+	if (lgr && mutex_is_locked(&lgr->conf_mutex))
+		mutex_unlock(&lgr->conf_mutex);
+	smc_sock_proc_create(newsmcsk);
+	sk_refcnt_debug_inc(newsmcsk);
+	if (!new_smc->use_tcp)
+		setup_timer(&newsmcsk->sk_timer, smc_keepalive_timer,
+			    (unsigned long)newsmcsk);
+	smc_init_after_clc_handshake(new_smc);
+enqueue:
+	if (lsmc->sk.sk_state == SMC_LISTEN) {
+		smc_accept_enqueue(&lsmc->sk, newsmcsk);
+	} else {
+		smc_destruct_non_accepted(newsmcsk);
+		new_smc = NULL;
+	}
+
+	/* Wake up accept */
+	lsmc->sk.sk_data_ready(&lsmc->sk);
+	sock_put(&lsmc->sk);
+	return;
+
+decline:
+	if (!smc_switch_to_tcp(new_smc, reason_code))
+		goto out_connected;
+
+out_err:
+	if (lgr && mutex_is_locked(&lgr->conf_mutex))
+		mutex_unlock(&lgr->conf_mutex);
+	smc_free_conn(new_smc);
+
+	if (!newsmcsk->sk_err)
+		newsmcsk->sk_err = EPROTO;
+	newsmcsk->sk_state = SMC_CLOSED;
+	if (newsmcsk->sk_err == ETIMEDOUT) {
+		/* forget the new socket */
+		if (new_smc->proc) {
+			proc_remove(new_smc->proc->proc);
+			new_smc->proc = NULL;
+		}
+		sock_set_flag(newsmcsk, SOCK_ZAPPED);
+		sock_set_flag(newsmcsk, SOCK_DEAD);
+		sock_release(newtcpsocket);
+		new_smc->tcpsocket = NULL;
+		sock_put(newsmcsk);
+		sock_put(&lsmc->sk);
+	} else {
+		goto enqueue;
+	}
+}
+
+static void smc_tcp_listen_worker(struct work_struct *work)
+{
+	struct smc_sock *lsmc = container_of(work, struct smc_sock,
+					tcp_listen_work);
+	struct sock *newsmcsk;
+	struct smc_sock *new_smc = NULL;
+	struct socket *newtcpsocket = NULL;
+	struct tcp_sock *ntp;
+	int rc = 0;
+
+	while (lsmc->sk.sk_state == SMC_LISTEN) {
+		rc = kernel_accept(lsmc->tcpsocket, &newtcpsocket, 0);
+		if (lsmc->sk.sk_state == SMC_CLOSED) {
+			if (newtcpsocket)
+				sock_release(newtcpsocket);
+			goto out;
+		}
+		if (rc) {
+			/* find out if there is an error we have to
+			 * restart tcp accept
+			 */
+			lsmc->sk.sk_err = -rc;
+			goto out;
+		}
+		ntp = tcp_sk(newtcpsocket->sk);
+
+		newsmcsk = smc_sock_alloc(NULL, PF_SMC, GFP_KERNEL);
+		if (!newsmcsk) {
+			lsmc->sk.sk_err = ENOMEM;
+			sock_release(newtcpsocket);
+			goto out;
+		}
+
+		new_smc = smc_sk(newsmcsk);
+		new_smc->tcpsocket = newtcpsocket;
+		new_smc->listen_smc = lsmc;
+		sock_hold(&lsmc->sk); /* sock_put in smc_listen_worker */
+		INIT_WORK(&new_smc->listen_work, smc_listen_worker);
+		smc_copy_sockopt_settings(newsmcsk, lsmc->tcpsocket->sk);
+		schedule_work(&new_smc->listen_work);
+	}
+
+out:
+	lsmc->sk.sk_data_ready(&lsmc->sk); /* wake up accept socket */
+}
+
+static int smc_sock_getname(struct socket *sock, struct sockaddr *addr,
+			    int *len, int peer)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc;
+
+	if (smc->tcpsocket)
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket, addr, len,
+						  peer);
+	else
+		rc = -EBADF;
+	return rc;
+}
+
+static int smc_sock_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int rc;
+
+	lock_sock(sk);
+	if (sk->sk_shutdown & SEND_SHUTDOWN) {
+		rc = -EPIPE;
+		goto out_pipe;
+	}
+
+	if (smc->use_tcp) {
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->sendmsg(smc->tcpsocket, msg,
+							  len);
+		else
+			rc = -EBADF;
+		goto out;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ECONNRESET;
+		goto out;
+	}
+	if (smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ENOTCONN;
+		goto out;
+	}
+	if (sk->sk_err) {
+		rc = -EPIPE;
+		goto out_pipe;
+	}
+
+	rc = smc_conn_send(smc, msg, len);
+out_pipe:
+	rc = sk_stream_error(sk, msg->msg_flags, rc);
+out:
+	release_sock(sk);
+
+	return rc;
+}
+
+static int smc_sock_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
+			    int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	lock_sock(sk);
+	if (smc->use_tcp) {
+		if (!smc->tcpsocket)
+			return -ENOTCONN;
+		rc = smc->tcpsocket->ops->recvmsg(smc->tcpsocket, msg, len,
+						  flags);
+	} else {
+		rc = smc_conn_recv(smc, msg, len, flags);
+	}
+	release_sock(sk);
+	return rc;
+}
+
+static unsigned int smc_accept_poll(struct sock *parent)
+{
+	struct smc_sock *isk;
+	struct sock *sk;
+
+	lock_sock(parent);
+	list_for_each_entry(isk, &smc_sk(parent)->accept_q, accept_q) {
+		sk = (struct sock *)isk;
+
+		if (sk->sk_state == SMC_ACTIVE) {
+			release_sock(parent);
+			return POLLIN | POLLRDNORM;
+		}
+	}
+	release_sock(parent);
+
+	return 0;
+}
+
+static unsigned int smc_sock_poll(struct file *file, struct socket *sock,
+				  poll_table *wait)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	unsigned int mask = 0;
+	int rc = 0;
+
+	if ((sk->sk_state == SMC_INIT) ||
+	    (smc->use_tcp && sk->sk_state != SMC_LISTEN)) {
+		if (smc->tcpsocket) {
+			mask = smc->tcpsocket->ops->poll(file, smc->tcpsocket,
+							 wait);
+			if ((sk->sk_state == SMC_INIT) && (mask & POLLOUT)) {
+				sk->sk_err = smc->tcpsocket->sk->sk_err;
+				if (sk->sk_err) {
+					mask |= POLLERR;
+				} else {
+					rc = smc_connect(smc);
+					if (rc)
+						mask |= POLLERR;
+					else
+						mask |= POLLOUT | POLLWRNORM;
+				}
+			}
+		} else {
+			sk->sk_err = EBADF;
+			mask |= POLLERR;
+		}
+	} else {
+		sock_poll_wait(file, sk_sleep(sk), wait);
+		if (sk->sk_state == SMC_LISTEN)
+			mask |= smc_accept_poll(sk);
+		if (sk->sk_err)
+			mask |= POLLERR;
+		if ((sk->sk_shutdown == SHUTDOWN_MASK) ||
+		    (sk->sk_state == SMC_CLOSED))
+			mask |= POLLHUP;
+		if (sk->sk_shutdown & RCV_SHUTDOWN)
+			mask |= POLLIN | POLLRDNORM | POLLRDHUP;
+		if (smc_to_read(smc))
+			mask |= POLLIN | POLLRDNORM;
+		if (sk->sk_state == SMC_APPLCLW1)
+			mask |= POLLIN;
+		if (!(sk->sk_shutdown & SEND_SHUTDOWN)) {
+			if (atomic_read(&smc->conn.tx_buf_space)) {
+				mask |= POLLOUT | POLLWRNORM;
+			} else {
+				set_bit(SOCK_ASYNC_NOSPACE,
+					&sk->sk_socket->flags);
+				set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+			}
+		} else {
+			mask |= POLLOUT | POLLWRNORM;
+		}
+		if ((smc->conn.rx_urg_state & SMC_URG_VALID) &&
+		    !(smc->conn.rx_urg_state & (SMC_URG_READ | SMC_URG_RECV)))
+			mask |= POLLPRI;
+	}
+	return mask;
+}
+
+static int smc_conn_shutdown(struct smc_sock *smc, int how)
+{
+	struct sock *sk = &smc->sk;
+	int rc = 0, ret = 0, old_state;
+
+	if (how == SHUT_RD)
+		return rc;
+
+	lock_sock(sk);
+	old_state = sk->sk_state;
+
+	if (how == SHUT_RDWR) {
+		smc_sk_release(smc, 1);
+		goto out;
+	}
+
+	/* how = SHUT_WR */
+	switch (sk->sk_state) {
+	/* Normal termination - Actice close parts */
+	case SMC_ACTIVE:
+		sk->sk_state = SMC_PEERCLW1;
+		/* fall through */
+	case SMC_PEERCLW1:
+		smc_wait_close(smc);
+		/* fall through */
+	case SMC_PEERCLW2:
+		/* if shutdown(both) and unread data then connection reset
+		 * if outstanding data to be written or RDMA not compl,
+		 * then mask conn as pending close
+		 */
+		smc->conn.local_tx_ctrl.conn_state_flags.sending_done = 1;
+		ret = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		break;
+	/* Normal termination - Passive close parts */
+	case SMC_APPLCLW1:
+		smc_wait_close(smc);
+		sk->sk_state = SMC_APPLCLW2;
+		smc->conn.local_tx_ctrl.conn_state_flags.sending_done = 1;
+		ret = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		break;
+	default:
+		break;
+	}
+
+out:
+	if (old_state != sk->sk_state)
+		sk->sk_state_change(&smc->sk);
+	release_sock(sk);
+	if (ret)
+		rc = -ENOTCONN;
+	return rc;
+}
+
+static int smc_sock_shutdown(struct socket *sock, int how)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (how > SHUT_RDWR)
+		return -EINVAL;
+
+	kernel_sock_shutdown(smc->tcpsocket, how);
+
+	if (!smc->use_tcp)
+		rc = smc_conn_shutdown(smc, how);
+	sk->sk_shutdown |= ++how;
+	return rc;
+}
+
+static u32 smc_keepalive_time_elapsed(struct smc_sock *smc)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+
+	lgr = smc->conn.lgr;
+	link = &lgr->lnk[smc->conn.lnk_idx];
+	return (u32)jiffies - link->rcv_tstamp;
+}
+
+static int smc_sock_setsockopt(struct socket *sock, int level, int optname,
+			       char __user *optval, unsigned int optlen)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct timer_list *cork_timer = &smc->cork_timer;
+	int val;
+	int valbool;
+	u32 ka_tim, elapsed = 0;
+	int rc = 0;
+
+	if (!smc->tcpsocket)
+		return -EBADF;
+	lock_sock(sk);
+	if (level != SOL_TCP)
+		goto fallback;
+	switch (optname) {
+	case SMC_KEEPALIVE:
+		if (get_user(val, (int __user *)optval))
+			return -EFAULT;
+		valbool = val ? 1 : 0;
+		rc = kernel_setsockopt(smc->tcpsocket, SOL_SOCKET,
+				       SO_KEEPALIVE, (char *)&val,
+				       sizeof(val));
+		if (rc || smc->use_tcp)
+			goto ret;
+		if (sk->sk_state == SMC_ACTIVE) {
+			ka_tim = tp->keepalive_time ? : TCP_KEEPALIVE_TIME;
+			if (valbool && !sock_flag(sk, SOCK_KEEPOPEN)) {
+				sk_reset_timer(sk, &sk->sk_timer,
+					       jiffies + (unsigned long)ka_tim);
+			} else if (!valbool) {
+				sk_stop_timer(sk, &sk->sk_timer);
+				cancel_work_sync(&smc->timer_work);
+			}
+		}
+		if (valbool)
+			sock_set_flag(sk, SOCK_KEEPOPEN);
+		else
+			sock_reset_flag(sk, SOCK_KEEPOPEN);
+		break;
+	case TCP_KEEPIDLE:
+		rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket, SOL_TCP,
+			optname, optval, optlen);
+		if (rc || smc->use_tcp)
+			goto ret;
+		if (get_user(val, (int __user *)optval))
+			return -EFAULT;
+		ka_tim = val;
+		if (sk->sk_state == SMC_ACTIVE) {
+			if (ka_tim && sock_flag(sk, SOCK_KEEPOPEN)) {
+				elapsed = smc_keepalive_time_elapsed(smc);
+				if (ka_tim > elapsed)
+					elapsed = ka_tim - elapsed;
+				sk_reset_timer(sk, &sk->sk_timer,
+					       jiffies + elapsed);
+			} else {
+				sk_stop_timer(sk, &sk->sk_timer);
+				cancel_work_sync(&smc->timer_work);
+			}
+		}
+		break;
+	case TCP_CORK:
+		if (optlen < sizeof(int))
+			return -EINVAL;
+		if (get_user(val, (int __user *)optval))
+			return -EFAULT;
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			smc->tmp_sockopts.cork = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_CORK;
+		} else {
+			rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket,
+					level, optname, optval, optlen);
+			if (!rc && !smc->use_tcp) {
+				if (!val && timer_pending(cork_timer)) {
+					del_timer_sync(cork_timer);
+					destroy_timer_on_stack(cork_timer);
+					smc_write_data(smc);
+				}
+			}
+		}
+		break;
+	case TCP_DEFER_ACCEPT:
+		if (optlen < sizeof(int))
+			return -EINVAL;
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			if (get_user(val, (int __user *)optval))
+				return -EFAULT;
+			smc->tmp_sockopts.defer_accept = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_DEFER_ACCEPT;
+		} else {
+			goto fallback;
+		}
+		break;
+	case TCP_NODELAY:
+		if (optlen < sizeof(int))
+			return -EINVAL;
+		if (get_user(val, (int __user *)optval))
+			return -EFAULT;
+		if (sk->sk_state == SMC_INIT || sk->sk_state == SMC_LISTEN) {
+			smc->tmp_sockopts.nodelay = val;
+			smc->tmp_sockopts.set |= SMC_SOCKOPTS_NODELAY;
+		} else {
+			if (!smc->use_tcp) {
+				if (val && timer_pending(cork_timer)) {
+					del_timer_sync(cork_timer);
+					destroy_timer_on_stack(cork_timer);
+					smc_write_data(smc);
+				}
+			}
+			goto fallback;
+		}
+		break;
+	default:
+		goto fallback;
+	}
+	goto ret;
+fallback:
+	rc = smc->tcpsocket->ops->setsockopt(smc->tcpsocket, level,
+		optname, optval, optlen);
+ret:
+	release_sock(sk);
+	return rc;
+}
+
+static int smc_sock_getsockopt(struct socket *sock, int level, int optname,
+			       char __user *optval, int __user *optlen)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int val, len;
+	int rc = 0;
+
+	if (smc->use_tcp)
+		if (!smc->tcpsocket)
+			return -EBADF;
+		return smc->tcpsocket->ops->getsockopt(smc->tcpsocket, level,
+						       optname, optval, optlen);
+	if (get_user(len, optlen))
+		return -EFAULT;
+	if (len < 0)
+		return -EINVAL;
+
+	if (level == SOL_IP) {
+		switch (optname) {
+		case IP_MTU:
+			if (sock->state != SS_CONNECTED || !smc->conn.lgr)
+				return -ENOTCONN;
+			val = ib_mtu_enum_to_int(
+				smc->conn.lgr->lnk[smc->conn.lnk_idx].mtu_peer);
+				break;
+		default:
+			goto fallback;
+		}
+	} else {
+		if (level != SOL_TCP)
+			goto fallback;
+	}
+	/* level is SOL_TCP */
+	switch (optname) {
+	case TCP_CORK:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = !!smc->tmp_sockopts.cork;
+		else
+			goto fallback;
+		break;
+	case TCP_DEFER_ACCEPT:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = smc->tmp_sockopts.defer_accept;
+		else
+			goto fallback;
+		break;
+	case TCP_NODELAY:
+		if (sk->sk_state == SMC_INIT ||
+		    sk->sk_state == SMC_LISTEN)
+			val = !!smc->tmp_sockopts.nodelay;
+		else
+			goto fallback;
+		break;
+	default:
+		goto fallback;
+	}
+
+	if (len < sizeof(val) && len > 0 && val >= 0 && val <= 255) {
+		unsigned char ucval = (unsigned char)val;
+
+		len = 1;
+		if (put_user(len, optlen))
+			return -EFAULT;
+		if (copy_to_user(optval, &ucval, 1))
+			return -EFAULT;
+	} else {
+		len = min_t(unsigned int, sizeof(int), len);
+		if (put_user(len, optlen))
+			return -EFAULT;
+		if (copy_to_user(optval, &val, len))
+			return -EFAULT;
+	}
+
+	return rc;
+
+fallback:
+	return smc->tcpsocket->ops->getsockopt(smc->tcpsocket, level,
+					       optname, optval, optlen);
+}
+
+static int smc_sock_ioctl(struct socket *sock, unsigned int cmd,
+			  unsigned long arg)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	struct smc_connection *conn = &smc->conn;
+	int answ;
+	struct smc_curs urg, c_curs;
+
+	if (smc->use_tcp) {
+		if (!smc->tcpsocket)
+			return -EBADF;
+		return smc->tcpsocket->ops->ioctl(smc->tcpsocket, cmd, arg);
+	}
+
+	switch (cmd) {
+	case SIOCINQ:
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		if (sk->sk_state == SMC_INIT)
+			answ = 0;
+		else
+			answ = smc_to_read(smc);
+		break;
+	case SIOCATMARK:
+		if ((sk->sk_state == SMC_INIT) ||
+		    (sk->sk_state == SMC_LISTEN)) {
+			answ = 0;
+		} else {
+			struct smc_e_ctrl *tx_ctrl = &conn->local_tx_ctrl;
+
+			urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+			if (!sock_flag(&smc->sk, SOCK_URGINLINE))
+				smc_curs_add(conn->rmb_rx_size, &urg.s.curs, 1);
+
+			c_curs.s.lcurs =
+				atomic64_read(&tx_ctrl->c_curs.s.acurs);
+			if ((conn->rx_urg_state & SMC_URG_VALID) &&
+			    (urg.s.curs.c == c_curs.s.curs.c) &&
+			    (urg.s.curs.w == c_curs.s.curs.w))
+				answ = 1;
+			else
+				answ = 0;
+		}
+		break;
+	case SIOCOUTQ:
+		/* output queue size (not send + not acked) */
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		if (sk->sk_state == SMC_INIT)
+			answ = 0;
+		else
+			answ = smc->conn.rmb_tx_size -
+					atomic_read(&smc->conn.tx_buf_space);
+		break;
+	case SIOCOUTQNSD:
+		/* output queue size (not send only) for smc always 0 */
+		if (sk->sk_state == SMC_LISTEN)
+			return -EINVAL;
+
+		answ = 0;
+		break;
+	default:
+		return -ENOIOCTLCMD;
+	}
+
+	return put_user(answ, (int __user *)arg);
+}
+
+static ssize_t smc_sock_sendpage(struct socket *sock, struct page *page,
+				 int offset, size_t size, int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (smc->use_tcp)
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->sendpage(smc->tcpsocket, page,
+							   offset, size, flags);
+		else
+			rc = -EBADF;
+	else
+		rc = sock_no_sendpage(sock, page, offset, size, flags);
+
+	return rc;
+}
+
+static ssize_t smc_sock_splice_read(struct socket *sock, loff_t *ppos,
+				    struct pipe_inode_info *pipe, size_t len,
+				    unsigned int flags)
+{
+	struct sock *sk = sock->sk;
+	struct smc_sock *smc = smc_sk(sk);
+	int rc = 0;
+
+	if (smc->use_tcp) {
+		if (smc->tcpsocket)
+			rc = smc->tcpsocket->ops->splice_read(smc->tcpsocket,
+							      ppos, pipe, len,
+							      flags);
+		else
+			rc = -EBADF;
+	} else {
+		if (*ppos)
+			/* we don't support offsets on our socket */
+			return -ESPIPE;
+		lock_sock(sk);
+		rc = smc_conn_splice_read(smc, pipe, len, flags);
+		release_sock(sk);
+	}
+
+	return rc;
+}
+
+/* must look like tcp */
+static const struct proto_ops smc_sock_ops = {
+	.family		= PF_SMC,
+	.owner		= THIS_MODULE,
+	.release	= smc_sock_release,
+	.bind		= smc_sock_bind,
+	.connect	= smc_sock_connect,
+	.socketpair	= sock_no_socketpair,
+	.accept		= smc_sock_accept,
+	.getname	= smc_sock_getname,
+	.poll		= smc_sock_poll,
+	.ioctl		= smc_sock_ioctl,
+	.listen		= smc_sock_listen,
+	.shutdown	= smc_sock_shutdown,
+	.setsockopt	= smc_sock_setsockopt,
+	.getsockopt	= smc_sock_getsockopt,
+	.sendmsg	= smc_sock_sendmsg,
+	.recvmsg	= smc_sock_recvmsg,
+	.mmap		= sock_no_mmap,
+	.sendpage	= smc_sock_sendpage,
+	.splice_read	= smc_sock_splice_read,
+};
+
+static int smc_sock_init_tcp(struct smc_sock *smc)
+{
+	int rc;
+
+	rc = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
+			      &smc->tcpsocket);
+	smc->sk.sk_rcvbuf = max_t(size_t, smc_def_rcvbuf_size,
+					  smc->tcpsocket->sk->sk_rcvbuf);
+	smc->sk.sk_sndbuf = max_t(size_t, smc_def_sndbuf_size,
+					  smc->tcpsocket->sk->sk_sndbuf);
+	tcp_sk(smc->tcpsocket->sk)->nonagle |= TCP_NAGLE_OFF | TCP_NAGLE_PUSH;
+	return rc;
+}
+
+static void smc_timer_worker(struct work_struct *);
+
+static void smc_free_proc_worker(struct work_struct *work)
+{
+	struct smc_proc_dir_entry *proc_to_free =
+		container_of(work, struct smc_proc_dir_entry, free_proc_work);
+	proc_remove(proc_to_free->proc);
+	kfree(proc_to_free);
+}
+
+static void smc_sock_destruct(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+
+	if (sk->sk_state != SMC_CLOSED)
+		return;
+	sk->sk_state = SMC_DESTRUCT;
+	if (!sock_flag(sk, SOCK_DEAD))
+		return;
+
+	if (smc->proc) {
+		smc->proc->sk = NULL;
+		schedule_work(&smc->proc->free_proc_work);
+		smc->proc = NULL;
+	}
+
+	smc_free_conn(smc);
+
+	sk_refcnt_debug_dec(sk);
+}
+
+static void smc_release_worker(struct work_struct *work)
+{
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+					release_work);
+
+	lock_sock(&smc->sk); /* make sure smc_sock_release is done */
+	if (!smc->use_tcp)
+		smc_wait_for_pending_sends(smc);
+	release_sock(&smc->sk);
+	sk_stop_timer(&smc->sk, &smc->sk.sk_timer);
+	cancel_work_sync(&smc->timer_work);
+	del_timer_sync(&smc->fin_timer);
+	destroy_timer_on_stack(&smc->fin_timer);
+	sock_put(&smc->sk);
+}
+
+static void smc_timer_worker(struct work_struct *work)
+{
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+					timer_work);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct llc_qentry *qentry;
+	int i, probes, rest_time, lnk_idx;
+
+	lgr = smc->conn.lgr;
+	probes = tp->keepalive_probes ? : TCP_KEEPALIVE_PROBES;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		link = &lgr->lnk[i];
+		lnk_idx = link - &lgr->lnk[0];
+		if (atomic_read(&link->state) != SMC_LINK_UP)
+			continue;
+		/* initiate keepalive */
+		rest_time = llc_initiate(lgr, LLC_GRP_TEST_LINK);
+		if (!rest_time) {
+			link->probes_out++;
+			if ((link->probes_out >= probes) &&
+			    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0)) {
+				schedule_work(&lgr->link_down_work);
+			}
+			return;
+		}
+		if (llc_send_test_link(link, "KEEPALIVE_CHECK")) {
+			llc_stop(lgr, LLC_GRP_TEST_LINK);
+			return;
+		}
+		qentry = llc_wait(lgr, LLC_TESTLINK, LLC_WAIT_TIMER_TIMEO,
+				  LLC_TEST_LINK);
+		if (!qentry) {
+			link->probes_out++;
+			if ((link->probes_out >= probes) &&
+			    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0)) {
+				schedule_work(&lgr->link_down_work);
+			}
+		} else {
+			link->probes_out = 0;
+		}
+		kfree(qentry);
+		llc_stop(lgr, LLC_GRP_TEST_LINK);
+	}
+}
+
+static void smc_keepalive_timer(unsigned long data)
+{	/* derived from tcp_keepalive_timer() */
+	struct sock *sk = (struct sock *)data;
+	struct smc_sock *smc = smc_sk(sk);
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct inet_connection_sock *icsk = inet_csk(smc->tcpsocket->sk);
+	u32 elapsed, time, intvl;
+
+	/* Only process if socket is not in use. */
+	bh_lock_sock(sk);
+	if (sock_owned_by_user(sk)) {
+		/* Try again later */
+		sk_reset_timer(sk, &sk->sk_timer, jiffies + (HZ / 20));
+		goto out;
+	}
+
+	if (!sock_flag(sk, SOCK_KEEPOPEN) || (sk->sk_state != SMC_ACTIVE))
+		goto out;
+
+	time = tp->keepalive_time ? : TCP_KEEPALIVE_TIME;
+	if (!time)
+		goto out;
+	intvl = tp->keepalive_intvl ? : TCP_KEEPALIVE_INTVL;
+	elapsed = smc_keepalive_time_elapsed(smc);
+
+	if ((elapsed >= intvl) &&
+	    (!icsk->icsk_user_timeout ||
+	     (elapsed >= icsk->icsk_user_timeout))) {
+		schedule_work(&smc->timer_work);
+		elapsed = intvl;
+	}
+
+	sk_reset_timer(sk, &sk->sk_timer, jiffies + (unsigned long)elapsed);
+out:
+	bh_unlock_sock(sk);
+	sock_put(sk); /* taken in sk_reset_timer() */
+}
+
+static void smc_corked_write_cb(unsigned long arg)
+{
+	struct smc_sock *smc = (struct smc_sock *)arg;
+
+	schedule_delayed_work(&smc->write_work, 0);
+}
+
+static void smc_fin_cb(unsigned long arg)
+{
+	struct smc_sock *smc = (struct smc_sock *)arg;
+	struct sock *tcpsk;
+
+	sock_hold(&smc->sk);
+	if (smc->sk.sk_state == SMC_CLOSED)
+		goto out;
+	smc->sk.sk_err = ETIMEDOUT;
+	smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+	if (smc->tcpsocket && smc->tcpsocket->sk) {
+		smc->tcpsocket->sk->sk_err = ECONNABORTED;
+		smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+	}
+	smc->sk.sk_state = SMC_CLOSED;
+	smc->sk.sk_shutdown = smc->sk.sk_shutdown | RCV_SHUTDOWN;
+	if (smc->tcpsocket) {
+		tcpsk = smc->tcpsocket->sk;
+		tcpsk->sk_shutdown = tcpsk->sk_shutdown | RCV_SHUTDOWN;
+	}
+	sock_set_flag(&smc->sk, SOCK_DEAD);
+	smc->sk.sk_state_change(&smc->sk);
+	wake_up_interruptible(&smc->rx_waiter);
+	smc_sock_wake_tx(&smc->sk);
+	smc_sock_wake_rx(&smc->sk);
+	if (cmpxchg(&smc->sock_put_done, 0, 1) == 0)
+		schedule_work(&smc->release_work);
+out:
+	sock_put(&smc->sk);
+}
+
+static struct sock *smc_sock_alloc(struct socket *sock, int proto, gfp_t prio)
+{
+	struct sock *sk;
+	struct smc_proc_dir_entry *pde = NULL;
+	struct smc_sock *smc;
+
+	pde = kzalloc(sizeof(*pde), GFP_KERNEL);
+	if (!pde)
+		return NULL;
+
+	sk = sk_alloc(&init_net, PF_SMC, prio, &smc_proto, 0);
+	if (!sk) {
+		kfree(pde);
+		return NULL;
+	}
+
+	pde->sk = sk;
+	smc = smc_sk(sk);
+	smc->tcpsocket = NULL;
+	smc->use_tcp = 0;
+	memset(&smc->conn, 0, sizeof(struct smc_connection));
+	smc->spd = NULL;
+	smc->proc = pde;
+
+	sock_init_data(sock, sk); /* sets sk_refcnt to 1 */
+	sk->sk_state = SMC_INIT;
+	INIT_LIST_HEAD(&smc->accept_q);
+	spin_lock_init(&smc->accept_q_lock);
+	sk->sk_destruct = smc_sock_destruct;
+	INIT_WORK(&smc->release_work, smc_release_worker);
+	INIT_WORK(&smc->timer_work, smc_timer_worker);
+	INIT_WORK(&smc->proc->free_proc_work, smc_free_proc_worker);
+	sock_reset_flag(sk, SOCK_ZAPPED);
+	sock_reset_flag(sk, SOCK_DONE);
+	sk->sk_protocol = proto;
+	init_waitqueue_head(&smc->splice_waiter);
+	setup_timer(&smc->cork_timer, smc_corked_write_cb, (unsigned long)smc);
+	setup_timer(&smc->fin_timer, smc_fin_cb, (unsigned long)smc);
+	smc_init_tmp_sockopts(smc);
+
+	return sk;
+}
+
+static int smc_sock_create(struct net *net, struct socket *sock, int protocol,
+			   int kern)
+{
+	struct sock *sk;
+	struct smc_sock *smc;
+	int rc = 0;
+
+	sock->state = SS_UNCONNECTED;
+
+	switch (sock->type) {
+	case SOCK_STREAM:
+		sock->ops = &smc_sock_ops;
+		break;
+	default:
+		return -ESOCKTNOSUPPORT;
+	}
+
+	sk = smc_sock_alloc(sock, protocol, GFP_KERNEL);
+	if (!sk)
+		return -ENOMEM;
+
+	smc = smc_sk(sk);
+	rc = smc_sock_init_tcp(smc);
+	if (rc) {
+		sk_free(sk);
+		smc = NULL;
+		goto out;
+	}
+	smc_sock_proc_create(sk);
+	setup_timer(&sk->sk_timer, smc_keepalive_timer, (unsigned long)sk);
+
+	sk_refcnt_debug_inc(sk);
+out:
+	return rc;
+}
+
+static const struct net_proto_family smc_sock_family_ops = {
+	.family	= AF_SMC,
+	.owner	= THIS_MODULE,
+	.create	= smc_sock_create,
+};
+
+static int __init af_smc_init(void)
+{
+	int rc;
+
+#ifdef CONFIG_PROC_FS
+	rc = smc_proc_init();
+	if (rc) {
+		rc = -EFAULT;
+		goto out;
+	}
+#else
+	pr_warn("smc requires /proc file system\n");
+	rc = -ENOENT;
+	goto out;
+#endif /* CONFIG_PROC_FS */
+
+	INIT_LIST_HEAD(&smc_ib_devices);
+	spin_lock_init(&smc_ib_dev_lock);
+	INIT_LIST_HEAD(&smc_lgr_list.list);
+	spin_lock_init(&smc_lgr_list.lock);
+	INIT_LIST_HEAD(&smc_lgr_freeing_list);
+	smc_lgr_list.create_conn_pending = 0;
+
+	rc = proto_register(&smc_proto, 0);
+	if (rc) {
+		pr_warn("smc proto_register fails with %d\n", rc);
+		goto out;
+	}
+
+	rc = sock_register(&smc_sock_family_ops);
+	if (rc) {
+		pr_warn("smc sock_register fails with %d\n", rc);
+		goto out_proto;
+	}
+
+	rc = ib_register_client(&smc_ib_client);
+	if (rc)
+		goto out_sock;
+
+	return 0;
+
+out_sock:
+	sock_unregister(PF_SMC);
+out_proto:
+	proto_unregister(&smc_proto);
+out:
+	return rc;
+}
+
+static void __exit af_smc_exit(void)
+{
+	ib_unregister_client(&smc_ib_client);
+	sock_unregister(PF_SMC);
+	proto_unregister(&smc_proto);
+	smc_proc_exit();
+}
+
+module_init(af_smc_init);
+module_exit(af_smc_exit);
+
+MODULE_AUTHOR("Ursula Braun <ursula.braun@de.ibm.com>");
+MODULE_DESCRIPTION("smc socket address family");
+MODULE_VERSION(VERSION);
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_NETPROTO(PF_SMC);
diff --git a/net/smc/af_smc.h b/net/smc/af_smc.h
new file mode 100644
index 0000000..1391cd8
--- /dev/null
+++ b/net/smc/af_smc.h
@@ -0,0 +1,706 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Definitions for the SMC module
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ */
+
+#include <linux/types.h>
+#include <net/tcp_states.h>
+
+extern struct list_head smc_ib_devices;
+extern struct ib_client smc_ib_client;
+
+struct smc_lgr_list_struct {
+	struct list_head list;
+	spinlock_t lock;	/* protects list of link groups */
+	u32 create_conn_pending;/* delay 2nd connection creation */
+};
+
+extern struct smc_lgr_list_struct smc_lgr_list; /* list of link groups */
+extern struct list_head smc_lgr_freeing_list; /* link groups to be freed */
+extern u32 smc_ctrl_buf_cnt;		/* # of ctrl buffers per link */
+extern u32 smc_max_conn_per_lgr;	/* max. # of connections per lgr */
+extern u32 smc_port_add_delay;		/* port add worker delay */
+extern atomic_t smc_lgr_num;		/* for creation of a unique lgr id */
+extern atomic_t smc_reconfiguring;	/* signal port adding / deleting */
+extern unsigned int smc_def_sndbuf_size;	/* minimum sndbuf size */
+extern unsigned int smc_def_rcvbuf_size;	/* minimum rcvbuf size */
+
+#define SMC_VERSION 0x10			/* first 4 bits */
+#define SMC_LOCAL_PEERID_RESET	"%%%%%%%%"
+
+#define SMC_MAX_PNET_ID_LEN	16
+
+#define SMC_FIRST_CONTACT 0x08			/* bit 4 */
+#define SMC_LINK_GROUP_OUT_OF_SYNCH 0x08	/* bit 4 */
+
+#define SMC_MAX_CQE 32768	/* max. # of completion queue elements */
+#define SMC_MAX_WRE 1024	/* max. # of work requests in flight per QP */
+#define SMC_MAX_POLL_CQE 10	/* max. # of compl. queue elements in 1 poll */
+#define SMC_MAX_SYM_LINKS 2	/* max. # of symmetric links */
+#define SMC_MAX_RMB 255		/* max. # of RMBs/connections per link group */
+#define SMC_MIN_RMBE_SIZE 16384	/* minimum size of an RMBE */
+#define SMC_MAX_PORT_ADD_DELAY 60	/* max. delay for port add (in secs) */
+
+#define SMC_RMBE_SIZE_VALUES	16
+
+#define SMC_LNK_EYE_CATCHER	"SMC_LINK"
+#define SMC_LNK_EYE_CATCH_LEN	8
+
+#define SMC_SIZE_OF_CTRL_DATA	44
+#define SMC_SIZE_OF_CTRL_BUF	sizeof(struct rmb_e_ctrl)
+
+#define SMC_URG_VALID		0x01
+#define SMC_URG_RECV		0x02
+#define SMC_URG_READ		0x04
+#define SMC_URG_MASK		0x03
+
+#define SMC_CLC_V1		0x10
+
+#define SMC_CLC_PROPOSAL	0x01
+#define SMC_CLC_ACCEPTANCE	0x02
+#define SMC_CLC_CONFIRMATION	0x03
+#define SMC_CLC_DECLINE		0x04
+
+#define SMC_CLC_BLOCK		0
+#define SMC_CLC_NONBLOCK	1
+
+#define SMC_QP_TIMEOUT			15 /* 4096 * 2 ** timeout usec */
+#define SMC_MIN_RNR_TIMER		5
+#define SMC_RETRY_CNT			7 /* 7: infinite */
+#define SMC_RNR_RETRY			7 /* 7: infinite */
+#define LLC_WAIT_FIRST_TIMEO		(5 * HZ)
+#define LLC_WAIT_TIMEO			(2 * HZ)
+#define LLC_WAIT_TIMER_TIMEO		HZ
+#define SMC_FREE_LGR_TIMEO		(600 * HZ)
+#define SMC_WAIT_PENDING_SENDS_TIMEO	(LLC_WAIT_TIMEO - 1 * HZ)
+#define SMC_WAIT_FREE_CTRL_BUF_TIMEO	HZ
+#define SMC_WAIT_RX_SPACE_TIMEO		(10 * HZ)
+#define SMC_WAIT_TX_SPACE_TIMEO		(10 * HZ)
+
+#define SMC_CLC_DEC_RUI		0x01000000  /* insufficient resources         */
+#define SMC_CLC_DEC_RIM		0x01010000  /* insufficient memory resources  */
+#define SMC_CLC_DEC_RIQ		0x01020000  /* insufficient QP resources      */
+#define SMC_CLC_DEC_TU		0x02000000  /* timeout                        */
+#define SMC_CLC_DEC_TPC		0x02010000  /* timeout w4 proposal            */
+#define SMC_CLC_DEC_TAS		0x02020000  /* timeout w4 accept              */
+#define SMC_CLC_DEC_TCC		0x02030000  /* timeout w4 confirm             */
+#define SMC_CLC_DEC_TCL		0x02040000  /* timeout w4 QP confirm          */
+#define SMC_CLC_DEC_CUC		0x03000000  /* configuration error            */
+#define SMC_CLC_DEC_CSLV	0x03010000  /* server lacks RoCE acc. to VLAN */
+#define SMC_CLC_DEC_CNIC	0x03020000  /* RNIC error                     */
+#define SMC_CLC_DEC_SU		0x04000000  /* synchronization error          */
+#define SMC_CLC_DEC_SFCB	0x04010000  /* first contact bit expected     */
+#define SMC_CLC_DEC_FORM	0x05000000  /* format error in received resp. */
+#define SMC_CLC_DEC_REPLY	0x06000000  /* reply to a received decline    */
+#define SMC_CLC_DEC_SEND	0x07000000  /* sending problem                */
+#define SMC_CLC_DEC_INT		0x99990000  /* internal error                 */
+
+#define SMC_LLC_DEL_NOLNK	0x00100000  /* Unknown Link ID (no link)      */
+#define SMC_LLC_DEL_NOLGR	0x00200000  /* Unknown Link Group             */
+
+#define SMC_SOCKOPTS_CORK		0x00000001
+#define SMC_SOCKOPTS_DEFER_ACCEPT	0x00000002
+#define SMC_SOCKOPTS_NODELAY		0x00000004
+
+#ifdef CONFIG_X86
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x01 /* cache_align_size 64 */
+#else
+#ifdef CONFIG_PPC
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x02 /* cache_align_size 128 */
+#else
+#define SMC_CACHE_ALIGN_SIZE_SHORT	0x03 /* cache_align_size 256 */
+#endif /* CONFIG_PPC */
+#endif /* CONFIG_X86 */
+
+#define smc_sk(__sk) ((struct smc_sock *)__sk)
+
+#define smc_lnk_downing(state) \
+	(atomic_cmpxchg(state, SMC_LINK_UP, SMC_LINK_DOWN) == SMC_LINK_UP)
+
+#define smc_stop_received(conn) \
+	(conn->local_rx_ctrl.conn_state_flags.sending_done || \
+	 conn->local_rx_ctrl.conn_state_flags.abnormal_close || \
+	 conn->local_rx_ctrl.conn_state_flags.closed_conn)
+
+#define smc_close_received(conn) \
+	(conn->local_rx_ctrl.conn_state_flags.abnormal_close || \
+	 conn->local_rx_ctrl.conn_state_flags.closed_conn)
+
+enum smc_role {		/* possible roles of a link group */
+	SMC_CLNT,	/* client */
+	SMC_SERV	/* server */
+};
+
+enum smc_state {	/* possible states of an SMC socket */
+	SMC_ACTIVE	= TCP_ESTABLISHED,	/* (1) connected */
+	SMC_INIT	= TCP_SYN_SENT,		/* (2) */
+	SMC_CLOSED	= TCP_CLOSE,		/* (7) */
+	SMC_LISTEN	= TCP_LISTEN,	/* (10)required for SO_ACCEPTCONN */
+	SMC_PEERCLW1	= 20,
+	SMC_PEERCLW2	= 21,
+	SMC_APPLCLW1	= 22,
+	SMC_APPLCLW2	= 23,
+	SMC_APPLFINCLW	= 24,
+	SMC_PEERFINCLW	= 25,
+	SMC_PEERABORTW	= 26,
+	SMC_PROCESSABORT = 27,
+	SMC_DESTRUCT	= 32
+};
+
+enum smc_link_state {	/* possible states of a link */
+	SMC_LINK_FREED,
+	SMC_LINK_DOWN,
+	SMC_LINK_ACTIVATING,
+	SMC_LINK_UP
+};
+
+enum llc_ctl_ind {	/* possible llc event control range */
+	LLC_CLNT,	/* as in smc_role */
+	LLC_SERV,	/* as in smc_role */
+	LLC_TESTLINK,
+	LLC_NO_NOTIFY
+};
+
+enum llc_msg_group {	/* classifies llc events */
+	LLC_GRP_NONE		= 0,
+	LLC_GRP_ADD_LINK	= 2,
+	LLC_GRP_DEL_LINK	= 4,
+	LLC_GRP_CONF_RKEY	= 6,
+	LLC_GRP_TEST_LINK	= 7
+};
+
+struct llc_ctl {	/* llc event data */
+	struct llc_qentry	*qentry;	/* llc event data */
+	enum llc_msg_group	active;		/* current llc event group */
+	int			lnk_idx;	/* active link for llc event */
+	int			ctrl_elem;	/* used ctrl buffer element */
+	int			wc_status;	/* status of sent llc event */
+};
+
+struct rmb_conn_state_flags {
+#if defined(__BIG_ENDIAN_BITFIELD)
+	u8	sending_done : 1;
+	u8	closed_conn : 1;
+	u8	abnormal_close : 1;
+	u8	reserved : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+	u8      reserved : 5;
+	u8      abnormal_close : 1;
+	u8      closed_conn : 1;
+	u8      sending_done : 1;
+#endif
+
+} __packed;
+
+struct rmb_producer_flags {
+#if defined(__BIG_ENDIAN_BITFIELD)
+	u8	 write_blocked : 1;	/* Writing Blocked, no rx buf space */
+	u8	 urg_data_pending : 1;	/* Urgent Data Pending */
+	u8	 urg_data_present : 1;	/* Urgent Data Present */
+	u8	 cons_curs_upd_req : 1;	/* cursor update requested */
+	u8	 last_msg_valid_req : 1;/* message replay due to failover */
+	u8	 reserved : 3;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+	u8       reserved : 3;
+	u8       last_msg_valid_req : 1;/* message replay due to failover */
+	u8       cons_curs_upd_req : 1; /* cursor update requested */
+	u8       urg_data_present : 1;  /* Urgent Data Present */
+	u8       urg_data_pending : 1;  /* Urgent Data Pending */
+	u8       write_blocked : 1;     /* Writing Blocked, no rx buf space */
+#endif
+} __packed;
+
+/* in host byte order */
+struct smc_cursor {		/* SMC cursor - an offset in an RMBE */
+	u8	reserved0;
+	u8	reserved;
+	u16	w;		/* window wrap sequence number */
+	u32	c;		/* cursor (= offset) part */
+} __aligned(8);
+
+struct smc_curs {		/* overlay for atomic cursor handling */
+	union {
+		struct smc_cursor	curs;
+		atomic64_t		acurs;
+		long long		lcurs;
+	} s;
+} __aligned(8);
+
+/* in host byte order */
+struct smc_e_ctrl {		/* RMBE control information */
+	u8				ctrl_type;	/* type = 0xFE */
+	u8				ctrl_len;	/* length = 44 */
+	u16				ctrl_seq;	/* connection seq # */
+	u32				ctrl_token;	/* alert_token */
+	struct smc_curs			p_curs;		/* producer cursor */
+	struct smc_curs			c_curs;		/* consumer cursor,
+							 * piggy backed ctrl
+							 */
+	struct rmb_producer_flags	p_flags;	/* conn. tx/rx status */
+	struct rmb_conn_state_flags	conn_state_flags; /* peer conn. status*/
+	u8				reserved[18];
+	u8				reserved1[4];	/* fill up to dw */
+} __packed __aligned(8);
+
+/* in network byte order */
+struct rmb_cursor {		/* SMC cursor */
+	u8	reserved0;
+	u8	reserved;
+	__be16	wwrap_seq;
+	__be32	count;
+} __aligned(8);
+
+struct rmb_curs {
+	union {
+		struct	rmb_cursor	curs;
+		atomic64_t		acurs;
+		long long		lcurs;
+	} r;
+} __aligned(8);
+
+/* in network byte order */
+struct rmb_e_ctrl {
+	u8				ctrl_type;
+	u8				ctrl_len;
+	__be16				ctrl_seq;
+	__be32				ctrl_token;
+	struct rmb_curs			p_curs;
+	struct rmb_curs			c_curs;
+	struct rmb_producer_flags	prod_flags;
+	struct rmb_conn_state_flags	conn_state_flags;
+	u8				reserved[18];
+	u8				reserved1[4];
+} __packed __aligned(8);
+
+enum lgr_type {		/* link status of a link group */
+	INIT,
+	SINGLE,			/* local and peer have just 1 ib-device */
+	ASYMMETRIC_LOCAL,	/* local has 1, peer has 2 ib-devices */
+	ASYMMETRIC_PEER,	/* local has 2, peer has 1 ib-device */
+	SYMMETRIC,		/* local and peer have 2 ib-devices */
+	NONE
+};
+
+struct pending_send {	/* control data for a pending send request */
+	struct smc_connection	*conn;		/* socket connection */
+	struct smc_curs		cursor;		/* tx RMBE cursor sent */
+	struct smc_curs		p_cursor;	/* rx RMBE cursor produced */
+	u64			wr_id_send;	/* work request id sent */
+	u16			ctrl_seq;	/* conn. tx sequence # */
+	u8			post_ctl;	/* llc event control index */
+	u8			reserved;
+};
+
+struct tx_sge {		/* scatter gather element for a send request */
+	u32 offset;
+	u32 len;
+};
+
+struct smc_rtoken {			/* address/key of remote RMB */
+	u64		vaddr[SMC_MAX_SYM_LINKS + 1];
+	u32		rkey[SMC_MAX_SYM_LINKS + 1];
+	atomic_t	active;			/* used / unused entry */
+	u8		link_id[SMC_MAX_SYM_LINKS + 1];		/* link ref. */
+	u8		reserved[7 - SMC_MAX_SYM_LINKS];
+};
+
+struct rmb_tx_addrs {			/* tx RMBE info */
+	struct list_head	list;
+	char			*rmb_tx_dma[SMC_MAX_SYM_LINKS + 1]; /* mapped */
+	char			*rmb_tx;	/* tx RMBE address */
+	u32			used;		/* currently used / unused */
+};
+
+struct rmb_rx_addrs {			/* rx RMBE info */
+	struct list_head	list;
+	char			*rmb_rx_dma[SMC_MAX_SYM_LINKS + 1]; /* mapped */
+	char			*rmb_rx;	/* rx RMBE address */
+	struct ib_mr		*mr_rx[SMC_MAX_SYM_LINKS + 1];	/* rkey */
+	u32			used;		/* currently used / unused */
+};
+
+struct smc_roce_defs {			/* ib-device infos for a link */
+	struct smc_ib_device	*ibdev;
+	union ib_gid		gid;
+	unsigned short		vlan;
+	char			mac[6];
+	u8			sgid_idx;
+	u8			port;
+};
+
+/* SMC Link - long living - for more than 1 SMC Connection */
+struct smc_link {
+	char			lnk_eyecatch[8];
+	struct smc_link_group	*lnk_grp;
+	struct ib_pd		*roce_pd;	/* IB protection domain,
+						 * unique for every ROCE QP
+						 */
+	struct ib_qp		*roce_qp;	/* IB queue pair */
+	struct ib_qp_attr	qp_attr;	/* IB queue pair attributes */
+	struct smc_roce_defs	roce_defs;	/* ib-device infos */
+	atomic_t		state;		/* link state */
+	enum ib_mtu		mtu_peer;	/* mtu size of peer */
+	enum ib_mtu		path_mtu;	/* used mtu */
+	u32			qp_num_peer;	/* QP # of peer */
+	char			mac_peer[6];	/* = gid[8:10] || gid[13:15] */
+	char			gid_peer[16];	/* gid of peer */
+	u8			link_id;	/* unique # within link group */
+
+	struct rmb_e_ctrl	*ctrl_buf_tx;	/* for send of WR_SENDs */
+	dma_addr_t		ctrl_dma_tx;	/* mapped addr for ctrl buff */
+	struct ib_send_wr	*send_wr;	/* send work requests area */
+	struct ib_mr		*mr_tx;		/* from ib_get_dma_mr */
+	struct ib_sge		*send_sge;	/* send scatter gather area */
+	struct pending_send	*pending;	/* in-flight sends area */
+	unsigned long		*pending_bit_mask; /* used ctrl_buf_tx elems  */
+	u64			wr_id_send;	/* seq # of last sent WQE */
+	u32			psn_initial;	/* starting send WQE seq # */
+	u32			send_wr_num;	/* # of send ctrl buffers */
+
+	struct rmb_e_ctrl	*ctrl_buf_rx;	/* for recv of WR_SENDs */
+	dma_addr_t		ctrl_dma_rx;	/* mapped addr for ctrl buff */
+	struct ib_recv_wr	*recv_wr;	/* recv work requests area */
+	struct ib_sge		*recv_sge;	/* recv scatter gather area */
+	u64			wr_id_recv;	/* seq # of last recv WQE */
+	u32			psn_peer;	/* starting recv WQE seq # */
+	u32			recv_wr_num;	/* # of recv ctrl buffers */
+	u32			rcv_tstamp;	/* time of last recv event */
+
+	wait_queue_head_t	wqe_waiter;	/* w4 free ctrl_buf_tx elem */
+	unsigned int		probes_out;	/* # of TESTLINK probes */
+};
+
+/* SMC Link Group */
+struct smc_link_group {
+	struct list_head	list;
+	enum smc_role		role;		/* client or server */
+	enum lgr_type		lgr_type;	/* link status for lgr */
+	u32			lgr_id;		/* unique lgr id */
+	__be32			daddr;		/* destination ip address */
+	__be32			subnet;	/* subnet mask */
+	u8			mask_len;	/* # of significant mask bits */
+	unsigned short		vlan;		/* vlan id of lgr */
+	char			pnet_id[SMC_MAX_PNET_ID_LEN];
+						/* pnet_id of ROCEs */
+	char			peer_peer_id[8];/* unique system id of peer */
+	u8			max_links;	/* allowed # of links (2) */
+	struct mutex		conf_mutex;	/* link added or removed */
+	struct rb_root		conns_all;	/* connection tree */
+	rwlock_t		conns_lock;	/* protects conns_all */
+	unsigned int		conns_num;	/* current # of connections */
+	struct list_head	rmb_tx_bufs[16]; /* tx buffers */
+	struct list_head	rmb_rx_bufs[16]; /* rx buffers */
+	struct smc_rtoken	rtok[SMC_MAX_RMB]; /* remote addr/key pairs */
+	struct smc_link		lnk[SMC_MAX_SYM_LINKS + 1]; /* links */
+	struct list_head	llc_event_q;	/* queue for llc events */
+	spinlock_t		llc_event_q_lock; /* protects llc_event_q */
+	struct llc_qentry	*delayed_q;	/* postpone llc event handl. */
+	struct work_struct	llc_event_work; /* llc event worker */
+	struct llc_ctl		llc_ctl[3];	/* llc event controller */
+	spinlock_t		llc_ctl_lock;	/* protects llc_ctl */
+	wait_queue_head_t	llc_waiter;	/* w4 next llc event */
+	struct workqueue_struct	*llc_wq;	/* work queue for llc events */
+	struct delayed_work	free_lgr_work;	/* delayed freeing of an lgr */
+	struct delayed_work	free_lgr_final_work; /* final lgr freeing */
+	struct smc_ib_device	*ibdev_up;	/* ib-device - new link */
+	struct work_struct	link_up_work;	/* adding link to lgr */
+	struct work_struct	link_down_work;	/* removing link from lgr */
+	struct work_struct	llc_add_link_work; /* llc add event worker */
+	struct work_struct	llc_del_link_work; /* llc del event worker */
+	struct work_struct	llc_conf_rkey_work;/* llc confirm rkey worker*/
+	struct work_struct	llc_del_rkey_work; /* llc del rkey worker */
+	atomic_t		link_num;	/* unique id last added link */
+	u32			lnk_up;		/* link up in progress */
+	u32			lnk_down[SMC_MAX_SYM_LINKS + 1];
+						/* link down in progress */
+	int			cache_align_size;	/* rdma_write align */
+	u8			port_up;	/* ib-device port - new link */
+	char			nwm_data[16];	/* network management data */
+	u8			nwm_flags;	/* network management flags */
+};
+
+struct smc_connection {
+	struct rb_node		alert_node;
+	struct smc_link_group	*lgr;		/* link group of connection */
+	u32			alert_token_local; /* created with custom logic
+						    * to address RMB and RMBE
+						    */
+	int			lnk_idx;	/* link currently used */
+
+	int			rtok_idx;	/* idx to peer rkey/addr */
+	int			peer_rx_buf_len; /* without eyecatcher */
+	u8			peer_tcp_conn_idx; /* from tcp handshake */
+
+	struct smc_e_ctrl	local_tx_ctrl;	/* preparation for sending */
+	struct smc_curs		local_tx_ctrl_fin; /* last completed target
+						    * offset
+						    */
+	struct smc_curs		tx_curs_prep;	/* tx - prepared data */
+	struct smc_curs		tx_curs_sent;	/* tx - sent data */
+	struct smc_curs		tx_curs_fin;	/* tx - confirmed by peer */
+	struct smc_curs		tx_urg_curs;	/* position of urgent byte */
+	atomic_t		tx_buf_space;	/* remaining space in tx rmb */
+	u16			tx_seq;		/* sequence # for WR SENDs */
+	u16			tx_seq_fin;	/* sequence # confirmed */
+	spinlock_t		send_lock;	/* protect wr_sends */
+	/* tx buffer */
+	struct rmb_tx_addrs	*rmb_tx_elem;	/* tx RMBE addresses */
+	int			rmb_tx_size;	/* tx RMBE size <== sock wmem */
+	int			rmb_tx_size_short;
+	int			tx_rmb_reused;	/* new / reused RMB */
+
+	struct smc_e_ctrl	local_rx_ctrl;	/* filled during event_handl. */
+	struct smc_curs		rx_curs_confirmed; /* confirmed to peer */
+	struct smc_curs		local_rx_spliced; /* pos of in-flight data */
+	struct smc_curs		rx_urg_curs;	/* cursor to urgent data */
+	u8			rx_urg_data;	/* 1 byte urgent data */
+	u8			rx_urg_state;	/* urg data avail / rcvd */
+	u16			ctrl_seq_rcvd;	/* seq # of last recv */
+	atomic_t		rx_buf_space;	/* remaining space in rx rmb */
+	atomic_t		bytes_to_rcv;	/* arrived data,
+						 * not yet received
+						 */
+	/* rx buffer */
+	struct rmb_rx_addrs	*rmb_rx_elem;
+	int			rmb_rx_size;	/* rx RMBE size <== sock rmem */
+	int			rmb_rx_size_short;
+	int			rx_rmb_reused;	/* new / reused TMB */
+	int			upd_cons_limit;	/* lower limit for consumer
+						 * cursor update
+						 */
+};
+
+struct smc_ib_device {				/* list of ib-devices */
+	struct list_head	list;
+	struct ib_device	*dev;
+	struct ib_device_attr	attr;		/* ib device attributes */
+	struct ib_port_attr	pattr[2];	/* ib dev. port attributes */
+	struct ib_event_handler	event_handler;	/* global ib_event handler */
+	struct ib_cq		*roce_cq_send;	/* send completion queue */
+	struct ib_cq		*roce_cq_recv;	/* recv completion queue */
+	struct tasklet_struct	recv_tasklet;
+	struct tasklet_struct	send_tasklet;
+	struct work_struct	port_err_work;	/* port error worker */
+	struct delayed_work	port_add_work;	/* port add worker */
+	u32			port_err[2];	/* port error triggered */
+	u32			port_add[2];	/* port add triggered */
+};
+
+/* eye catcher "SMCR" EBCDIC */
+static const char SMC_EC[] = {0xe2, 0xd4, 0xc3, 0xd9};
+
+struct smc_clc_msg_hdr {	/* header1 of clc messages */
+	u8 ec_clce[4];		/* eye catcher */
+	u8 type;		/* proposal / accept / confirm / decline */
+	__be16 length;
+	u8 flags;
+} __packed;
+
+struct smc_clc_msg_hdr2 {	/* header2 of clc messages */
+	u8 peer_id[8];		/* unique system id */
+	union ib_gid gid;
+	u8 mac[6];
+} __packed;
+
+struct smc_proposal_clc_msg {	/* proposal clc message */
+	struct smc_clc_msg_hdr hdr;
+	struct smc_clc_msg_hdr2 hdr2;
+	__u16 iparea_offset;	/* offset to IP address information area */
+} __packed;
+
+struct smc_proposal_clc_msg2 {
+	__be32 outgoing_subnet;	/* subnet mask */
+	u8 subnet_mask_bits;	/* number of significant bits in mask */
+	u8 reserved[2];
+	u8 n;			/* number of IPv6 prefixes in prefix array */
+	u8 ec_pclcf[4];		/* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_acc_conf_clc_msg {	/* accept / confirm clc message */
+	struct smc_clc_msg_hdr hdr;
+	struct smc_clc_msg_hdr2 hdr2;
+	u8 qp_num[3];		/* QP number */
+	__be32 rmb_rkey;	/* RMB rkey */
+	u8 tcp_conn_idx;	/* TCP connection index, which RMBE in RMB */
+	__be32 rmb_alert_token; /* unique connection id */
+	u8 flags2;		/* server's RMB buf size (compressed notation)
+				 * + QP mtu
+				 */
+	u8 reserved;
+	__be64 rmb_vaddr;	/* RMB virtual address */
+	u8 reserved2;
+	u8 psn[3];		/* initial packet sequence number */
+	u8 ec_clcf[4]; /* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_decline_clc_msg {
+	struct smc_clc_msg_hdr hdr;
+	u8 peer_id[8];		/* sender peer_id */
+	u32 peer_diagnosis;	/* diagnosis information */
+	u8 reserved2[4];
+	u8 ec_dclcf[4]; /* eye catcher "SMCR" EBCDIC */
+} __packed;
+
+struct smc_tmp_sockopts {
+	unsigned int	set;	/* indicates what sockopts were user-set */
+	int		cork;
+	int		defer_accept;
+	int		nodelay;
+};
+
+struct smc_proc_dir_entry {
+	struct sock *sk;
+	struct proc_dir_entry *proc;
+	struct work_struct free_proc_work;
+};
+
+struct smc_sock {				/* smc socket */
+	struct sock sk;
+	struct socket *tcpsocket;		/* internal tcp socket */
+	struct smc_connection conn;		/* connection */
+	struct sockaddr *addr;			/* inet connect address */
+	bool new_lgr;				/* new lgr created for conn. */
+	bool use_tcp;				/* fallback to tcp */
+	struct smc_sock *listen_smc;		/* listen parent */
+	struct work_struct tcp_listen_work;	/* handle tcp socket accepts */
+	struct work_struct listen_work;		/* prepare new accept socket */
+	struct list_head	accept_q;	/* sockets to be accepted */
+	spinlock_t		accept_q_lock;	/* protects accept_q */
+	struct work_struct release_work;	/* w4 finished sents before
+						 * closing socket finally
+						 */
+	wait_queue_head_t prep_waiter;		/* w4 prep sent when closing */
+	wait_queue_head_t destruct_waiter;	/* w4 pending sends */
+	struct timer_list	fin_timer;	/* ensure socket closing even
+						 * if peer is gone
+						 */
+	u32 sock_put_done;			/* allow race free closing */
+	struct work_struct timer_work;		/* keepalive TESTLINK sending */
+	struct delayed_work write_work;		/* RDMA writes of tx data */
+	wait_queue_head_t rx_waiter;		/* w4 space in rx RMBE */
+	wait_queue_head_t	splice_waiter;
+	struct splice_pipe_desc *spd;
+	struct timer_list	cork_timer;
+	struct smc_tmp_sockopts tmp_sockopts;
+	struct smc_proc_dir_entry *proc;
+	char proc_name[48];			/* sockname in /proc/net/smc */
+};
+
+int smc_create_conn(struct smc_sock *, __be32, enum smc_role,
+		    struct smc_roce_defs *, struct smc_clc_msg_hdr2 *, u32);
+void smc_find_roce_resources(struct smc_roce_defs *, char *,
+			     struct sockaddr_in *, struct sock *);
+void smc_find_alt_roce_resources(struct smc_roce_defs *,
+				 struct smc_link_group *, struct smc_link *);
+int smc_gid_by_dev(struct smc_roce_defs *);
+int smc_netinfo_by_tcpsk(__be32 *, u8 *, struct socket *);
+int smc_pnet_by_ibdev(u8, char *, struct smc_ib_device *);
+int smc_get_ib_mac(struct smc_roce_defs *);
+int smc_create_link(struct smc_link_group *, int, struct smc_roce_defs *);
+int smc_port_active(struct smc_ib_device *, u8);
+int smc_ready_link(struct smc_link *);
+void smc_link_up(struct smc_link_group *);
+void smc_link_down(struct smc_link_group *, int);
+int smc_find_rtoken_by_link(struct smc_link_group *, int, u32);
+int smc_modify_qp_rts(struct smc_link *);
+int smc_modify_qp_reset(struct smc_link *);
+int clc_wait_msg(struct smc_sock *, int, char *, int, u8, int *);
+int clc_send_decline(struct smc_sock *, u32, u8);
+
+int smc_create_rmbs(struct smc_sock *);
+int smc_del_rmbs(struct smc_sock *);
+int smc_map_rmbs_tx(struct smc_link_group *, int, int,
+		    struct rmb_tx_addrs *);
+int smc_map_rmbs_rx(struct smc_link_group *, int, int,
+		    struct rmb_rx_addrs *);
+long smc_get_dma_mr(struct smc_link_group *, int, struct rmb_rx_addrs *);
+long smc_map_rmbs(struct smc_sock *);
+long smc_map_rmbs_to_link(struct smc_link_group *, int, int);
+int smc_get_rtoken(struct smc_link_group *);
+u8 smc_compress_bufsize(int);
+int smc_uncompress_bufsize(u8);
+
+int smc_conn_send(struct smc_sock *, struct msghdr *, size_t);
+int smc_conn_recv(struct smc_sock *, struct msghdr *, size_t, int);
+int smc_to_read(struct smc_sock *);
+int smc_get_ctrl_buf(struct smc_link *, u32 *);
+int smc_wr_send(struct smc_link *, struct smc_connection *, u64, u32);
+int smc_wait_rx_data(struct smc_sock *, int, bool, long);
+void smc_sock_wake_tx(struct sock *);
+void smc_sock_wake_rx(struct sock *);
+void smc_send_tasklet_fn(unsigned long);
+void smc_recv_tasklet_fn(unsigned long);
+ssize_t smc_conn_splice_read(struct smc_sock *, struct pipe_inode_info *,
+			     size_t, unsigned int);
+void smc_write_worker(struct work_struct *);
+void smc_write_data(struct smc_sock *);
+
+int smc_check_peer_gone(struct smc_link_group *);
+void smc_free_conn(struct smc_sock *);
+void smc_remove_rx_rmb(struct smc_sock *);
+void smc_free_link(struct smc_link_group *, int);
+void smc_free_lgr(struct smc_link_group *);
+int smc_wait_no_pending_sends_on_link(struct smc_link *);
+int smc_ctrl_send(struct smc_connection *, int);
+int smc_send_close(struct smc_sock *);
+void smc_send_close_alt(struct smc_sock *);
+int smc_switch_conns(struct smc_link_group *, int, int);
+void smc_terminate_conn(struct smc_link_group *);
+void smc_conn_term(struct smc_sock *);
+int smc_prepared_sends(struct smc_connection *);
+int smc_pending_sends(struct smc_connection *);
+void smc_clear_pending_sends(struct smc_connection *);
+
+int smc_sock_proc_create(struct sock *);
+void smc_sock_proc_remove(char *);
+int smc_proc_init(void);
+void smc_proc_exit(void);
+void smc_check_dev_attr(struct smc_ib_device *);
+void smc_check_port_attr(struct smc_ib_device *, int);
+void smc_check_qp_attr(struct smc_link *);
+void smc_cq_handler_recv(struct ib_cq *, void *);
+void smc_cq_handler_send(struct ib_cq *, void *);
+void smc_cq_event_handler(struct ib_event *, void *);
+
+static inline void hton_three(u32 host, u8 *net)
+{
+	u32 t;
+
+	t = cpu_to_be32(host);
+	memcpy(net, ((u8 *)&t) + 1, 3);
+}
+
+static inline void ntoh_three(u32 *host, u8 *net)
+{
+	u32 t = 0;
+
+	memcpy(((u8 *)&t) + 1, net, 3);
+	*host = be32_to_cpu(t);
+}
+
+static inline void smc_curs_dec(int size, struct smc_cursor *curs)
+{
+	if (!curs->c)
+		curs->w--;
+	curs->c = (curs->c + size - 1) % size;
+}
+
+static inline void smc_curs_add(int size, struct smc_cursor *curs, int value)
+{
+	curs->w += (curs->c + value) / size;
+	curs->c = (curs->c + value) % size;
+}
+
+/* calculate cursor difference between old and new, where old <= new */
+static inline int smc_curs_diff(unsigned int size, struct smc_curs *old,
+				struct smc_curs *new)
+{
+	if (old->s.curs.w != new->s.curs.w)
+		return max_t(int, 0, ((size - old->s.curs.c) + new->s.curs.c));
+	return max_t(int, 0, (new->s.curs.c - old->s.curs.c));
+}
diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
new file mode 100644
index 0000000..c7f9504
--- /dev/null
+++ b/net/smc/smc_core.c
@@ -0,0 +1,3291 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Basic Transport Functions exploiting Infiniband API
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              Stefan Raspl <raspl@linux.vnet.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/wait.h>
+#include <linux/kthread.h>
+#include <linux/hardirq.h>
+#include <net/tcp.h>
+#include <net/sock.h>
+#include <linux/types.h>
+#ifdef CONFIG_S390
+#include <asm/debug.h>
+#endif
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <linux/splice.h>
+#include <linux/mm.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+#define LGR_RECONFIGURING 1
+
+/* Find the connection associated with the given alert token in the link group.
+ * Requires @conns_lock.
+ * @token	alert token to search for
+ * @lgr		 link group to search in
+ * Returns connection associated with token if found, NULL otherwise.
+ */
+static inline
+struct smc_connection *smc_find_conn(u32 token, struct smc_link_group *lgr)
+{
+	struct rb_node *node;
+	struct smc_connection *res = NULL;
+
+	node = lgr->conns_all.rb_node;
+	while (node) {
+		struct smc_connection *cur = rb_entry(node,
+					struct smc_connection, alert_node);
+
+		if (cur->alert_token_local > token) {
+			node = node->rb_left;
+		} else {
+			if (cur->alert_token_local < token) {
+				node = node->rb_right;
+			} else {
+				res = cur;
+				break;
+			}
+		}
+	}
+
+	return res;
+}
+
+/* Register connection's alert token in our lookup structure.
+ * Requires @conns_lock.
+ * @smc		connection to register
+ * Returns 0 on success, != otherwise.
+ */
+static void _smc_register_alert_token(struct smc_connection *conn)
+{
+	struct rb_node **link, *parent = NULL;
+	u32 token = conn->alert_token_local;
+
+	link = &conn->lgr->conns_all.rb_node;
+	while (*link) {
+		struct smc_connection *cur = rb_entry(*link,
+					struct smc_connection, alert_node);
+
+		parent = *link;
+		if (cur->alert_token_local > token)
+			link = &parent->rb_left;
+		else
+			link = &parent->rb_right;
+	}
+	/* Put the new node there */
+	rb_link_node(&conn->alert_node, parent, link);
+	rb_insert_color(&conn->alert_node, &conn->lgr->conns_all);
+}
+
+/* Register connection in link group by assigning an alert token
+ * registered in a search tree.
+ * Requires lgr->conns_lock being held.
+ * Note that '0' is a reserved value and not assigned.
+ */
+static void smc_register_in_lgr(struct smc_connection *conn)
+{
+	static atomic_t nexttoken = ATOMIC_INIT(0);
+
+	WARN_ON(conn->alert_token_local);
+	while (!conn->alert_token_local) {
+		conn->alert_token_local = atomic_inc_return(&nexttoken);
+		if (smc_find_conn(conn->alert_token_local, conn->lgr))
+			conn->alert_token_local = 0;
+	}
+	_smc_register_alert_token(conn);
+	conn->lgr->conns_num++;
+}
+
+/* Unregister and reset the alert token of the given connection
+ */
+static void smc_unregister_in_lgr(struct smc_connection *conn)
+{
+	struct smc_link_group *lgr = conn->lgr;
+
+	if (conn->alert_token_local) {
+		write_lock_bh(&lgr->conns_lock);
+		rb_erase(&conn->alert_node, &lgr->conns_all);
+		lgr->conns_num--;
+		write_unlock_bh(&lgr->conns_lock);
+	}
+}
+
+static void smc_sock_wake_rx_for_tx(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+	struct socket *sock = sk->sk_socket;
+
+	if (sock && (sock->state == SS_CONNECTED) &&
+	    atomic_read(&smc->conn.rx_buf_space))
+		wake_up_interruptible(&smc->rx_waiter);
+}
+
+static void smc_free_rmbs(struct smc_connection *conn)
+{
+	int i;
+
+	if (conn->rmb_tx_elem) {
+		i = ror8(conn->rmb_tx_size_short, 4);
+		cmpxchg(&conn->rmb_tx_elem->used, 1, 0);
+	}
+	conn->rmb_tx_elem = NULL;
+	if (conn->rmb_rx_elem) {
+		i = ror8(conn->rmb_rx_size_short, 4);
+		cmpxchg(&conn->rmb_rx_elem->used, 1, 0);
+	}
+	conn->rmb_rx_elem = NULL;
+}
+
+static int smc_find_same_lnk(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (i == lnk_idx)
+			continue;
+		if ((lgr->lnk[i].roce_defs.ibdev ==
+			lgr->lnk[lnk_idx].roce_defs.ibdev) &&
+		    (lgr->lnk[i].roce_defs.port ==
+			lgr->lnk[lnk_idx].roce_defs.port))
+			return i;
+	}
+	return i;
+}
+
+static void smc_unmap_rmbs(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+	struct rmb_tx_addrs *txkp;
+	struct rmb_rx_addrs *rxkp;
+	struct smc_roce_defs *rocdefs = &lgr->lnk[lnk_idx].roce_defs;
+	int reuse_lnk_idx;
+
+	reuse_lnk_idx = smc_find_same_lnk(lgr, lnk_idx);
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(txkp, &lgr->rmb_tx_bufs[i], list) {
+			if (txkp->rmb_tx_dma[lnk_idx] &&
+			    (reuse_lnk_idx > SMC_MAX_SYM_LINKS))
+				ib_dma_unmap_single(rocdefs->ibdev->dev,
+					(dma_addr_t)txkp->rmb_tx_dma[lnk_idx],
+					smc_uncompress_bufsize(rol8(i, 4)),
+					DMA_TO_DEVICE);
+			txkp->rmb_tx_dma[lnk_idx] = NULL;
+		}
+		list_for_each_entry(rxkp, &lgr->rmb_rx_bufs[i], list) {
+			if (rxkp->rmb_rx_dma[lnk_idx] &&
+			    (reuse_lnk_idx > SMC_MAX_SYM_LINKS))
+				ib_dma_unmap_single(rocdefs->ibdev->dev,
+					(dma_addr_t)rxkp->rmb_rx_dma[lnk_idx],
+					smc_uncompress_bufsize(rol8(i, 4)),
+					DMA_FROM_DEVICE);
+			rxkp->rmb_rx_dma[lnk_idx] = NULL;
+			if (rxkp->mr_rx[lnk_idx])
+				ib_dereg_mr(rxkp->mr_rx[lnk_idx]);
+			rxkp->mr_rx[lnk_idx] = NULL;
+		}
+	}
+}
+
+static void smc_clear_rtokens(struct smc_link_group *lgr, int lnk_idx)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		lgr->rtok[i].rkey[lnk_idx] = 0;
+		lgr->rtok[i].vaddr[lnk_idx] = 0;
+		lgr->rtok[i].link_id[lnk_idx] = 0;
+	}
+}
+
+static void smc_free_link_mem(struct smc_link *lnk)
+{
+	kfree(lnk->pending);
+	lnk->pending = NULL;
+	kfree(lnk->pending_bit_mask);
+	lnk->pending_bit_mask = NULL;
+	kfree(lnk->send_sge);
+	lnk->send_sge = NULL;
+	kfree(lnk->recv_sge);
+	lnk->recv_sge = NULL;
+	kfree(lnk->recv_wr);
+	lnk->recv_wr = NULL;
+	kfree(lnk->send_wr);
+	lnk->send_wr = NULL;
+	kfree(lnk->ctrl_buf_tx);
+	lnk->ctrl_buf_tx = NULL;
+	kfree(lnk->ctrl_buf_rx);
+	lnk->ctrl_buf_rx = NULL;
+}
+
+static int smc_modify_qp_init(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_INIT;
+	qp_attr.pkey_index = 0;
+	qp_attr.port_num = link->roce_defs.port;
+	qp_attr.qp_access_flags = IB_ACCESS_LOCAL_WRITE
+				| IB_ACCESS_REMOTE_WRITE;
+	rc = ib_modify_qp(link->roce_qp, &qp_attr,
+			  IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_ACCESS_FLAGS |
+			  IB_QP_PORT);
+	return rc;
+}
+
+static int smc_modify_qp_rtr(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+	enum ib_qp_attr_mask qp_attr_mask =
+		IB_QP_STATE | IB_QP_AV | IB_QP_PATH_MTU | IB_QP_DEST_QPN |
+		IB_QP_RQ_PSN | IB_QP_MAX_DEST_RD_ATOMIC | IB_QP_MIN_RNR_TIMER |
+		IB_QP_SMAC;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RTR;
+	qp_attr.path_mtu = min(link->path_mtu, link->mtu_peer);
+	qp_attr.ah_attr.port_num = link->roce_defs.port;
+	qp_attr.ah_attr.ah_flags = IB_AH_GRH;
+	qp_attr.ah_attr.grh.hop_limit = 1;
+	memcpy(&qp_attr.ah_attr.grh.dgid, link->gid_peer,
+	       sizeof(link->gid_peer));
+	qp_attr.ah_attr.grh.sgid_index = link->roce_defs.sgid_idx;
+	if (link->roce_defs.vlan) {
+		qp_attr.ah_attr.vlan_id = link->roce_defs.vlan;
+		qp_attr.vlan_id = link->roce_defs.vlan;
+		qp_attr_mask |= IB_QP_VID;
+	}
+	memcpy(&qp_attr.ah_attr.dmac, link->mac_peer,
+	       sizeof(link->mac_peer));
+	qp_attr.dest_qp_num = link->qp_num_peer;
+	qp_attr.rq_psn = link->psn_peer; /* starting receive packet seq # */
+	qp_attr.max_dest_rd_atomic = 1;  /* max # of resources for incoming
+					  * requests
+					  */
+	qp_attr.min_rnr_timer = SMC_MIN_RNR_TIMER;
+	memcpy(&qp_attr.smac, link->roce_defs.mac,
+	       sizeof(link->roce_defs.mac));
+
+	rc = ib_modify_qp(link->roce_qp, &qp_attr, qp_attr_mask);
+	return rc;
+}
+
+int smc_modify_qp_rts(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RTS;
+	qp_attr.timeout = SMC_QP_TIMEOUT;	/* local ack timeout */
+	qp_attr.retry_cnt = SMC_RETRY_CNT;	/* retry count */
+	qp_attr.rnr_retry = SMC_RNR_RETRY;	/* RNR retries, 7=infinite */
+	qp_attr.sq_psn = link->psn_initial;	/* starting send packet seq # */
+	qp_attr.max_rd_atomic = 1;	/* # of outstanding RDMA reads and
+					 * atomic ops allowed
+					 */
+	rc = ib_modify_qp(link->roce_qp, &qp_attr,
+			  IB_QP_STATE | IB_QP_TIMEOUT | IB_QP_RETRY_CNT |
+			  IB_QP_SQ_PSN | IB_QP_RNR_RETRY |
+			  IB_QP_MAX_QP_RD_ATOMIC);
+	return rc;
+}
+
+int smc_modify_qp_reset(struct smc_link *link)
+{
+	struct ib_qp_attr qp_attr;
+	int rc = 0;
+
+	memset(&qp_attr, 0, sizeof(qp_attr));
+	qp_attr.qp_state = IB_QPS_RESET;
+	rc = ib_modify_qp(link->roce_qp, &qp_attr, IB_QP_STATE);
+	return rc;
+}
+
+static inline int smc_outstanding_post_recv(struct smc_link *link)
+{
+	int rc = 0;
+	struct ib_recv_wr *bad_recv_wr = NULL;
+	u32 index;
+	u64 wr_id;
+
+	wr_id = link->wr_id_recv++; /* tasklet context, thus not atomic */
+	index = wr_id % link->recv_wr_num;
+	link->recv_wr[index].wr_id = wr_id;
+	rc = ib_post_recv(link->roce_qp, &link->recv_wr[index], &bad_recv_wr);
+	return rc;
+}
+
+int smc_ready_link(struct smc_link *link)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	int rc = 0;
+	u32 i;
+
+	rc = smc_modify_qp_init(link);
+	if (rc)
+		goto out;
+
+	rc = smc_modify_qp_rtr(link);
+	if (rc)
+		goto out;
+	smc_check_qp_attr(link);
+	rc = ib_req_notify_cq(link->roce_defs.ibdev->roce_cq_recv,
+			      IB_CQ_SOLICITED_MASK);
+	if (rc)
+		goto out;
+	for (i = 0; i < link->recv_wr_num; i++)
+		rc = smc_outstanding_post_recv(link);
+	smc_check_qp_attr(link);
+
+	if (lgr->role == SMC_SERV) {
+		rc = smc_modify_qp_rts(link);
+		if (rc)
+			goto out;
+		smc_check_qp_attr(link);
+	}
+out:
+	return rc;
+}
+
+static void smc_free_spd(struct smc_sock *);
+
+/* reset rx_rmb if rkey exchange fails */
+void smc_remove_rx_rmb(struct smc_sock *smc)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct rmb_rx_addrs *rxkp = conn->rmb_rx_elem;
+	int lnk_idx = conn->lnk_idx;
+	struct smc_link *lnk = &conn->lgr->lnk[lnk_idx];
+	struct smc_roce_defs *rocdefs = &lnk->roce_defs;
+
+	if (rxkp->mr_rx[lnk_idx])
+		ib_dereg_mr(rxkp->mr_rx[lnk_idx]);
+	if (rxkp->rmb_rx_dma[lnk_idx])
+		ib_dma_unmap_single(rocdefs->ibdev->dev,
+				    (dma_addr_t)rxkp->rmb_rx_dma[lnk_idx],
+				    conn->rmb_rx_size, DMA_FROM_DEVICE);
+	rxkp->rmb_rx_dma[lnk_idx] = NULL;
+	list_del(&rxkp->list);
+	kfree(rxkp);
+	conn->rmb_rx_elem = NULL;
+}
+
+void smc_free_conn(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int empty;
+
+	if (lgr) {
+		/* can be called from sk_free in atomic context */
+		smc_clear_pending_sends(&smc->conn);
+		smc_free_rmbs(&smc->conn);
+		smc_unregister_in_lgr(&smc->conn);
+		empty = RB_EMPTY_ROOT(&smc->conn.lgr->conns_all);
+		if (empty)
+			schedule_delayed_work(&lgr->free_lgr_work,
+					      SMC_FREE_LGR_TIMEO);
+		smc->conn.lgr = NULL;
+	}
+	if (smc->spd)
+		smc_free_spd(smc);
+}
+
+void smc_free_link(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct smc_link *lnk = &lgr->lnk[lnk_idx];
+	struct smc_ib_device *ibdev = lnk->roce_defs.ibdev;
+	enum smc_link_state old_state;
+
+	old_state = atomic_cmpxchg(&lnk->state, SMC_LINK_DOWN, SMC_LINK_FREED);
+	if (old_state != SMC_LINK_DOWN) {
+		if (old_state == SMC_LINK_FREED) /* link freeing in progress */
+			while (lnk->roce_defs.ibdev)
+				msleep(10000);
+		return;
+	}
+	lnk->link_id = 0;
+	memset(lnk->lnk_eyecatch, 0, SMC_LNK_EYE_CATCH_LEN);
+	memset(lnk->pending_bit_mask, 0,
+	       BITS_TO_LONGS(smc_ctrl_buf_cnt) * sizeof(unsigned long));
+	if (!lnk->roce_defs.ibdev)
+		return;
+	if (lnk->roce_qp) {
+		ib_destroy_qp(lnk->roce_qp);
+		lnk->roce_qp = NULL;
+	}
+	smc_unmap_rmbs(lgr, lnk_idx);
+	memset(&lnk->roce_defs, 0, sizeof(struct smc_roce_defs));
+	lnk->qp_num_peer = 0;
+	smc_clear_rtokens(lgr, lnk_idx);
+	if (lnk->mr_tx) {
+		ib_dereg_mr(lnk->mr_tx);
+		lnk->mr_tx = NULL;
+	}
+	if (lnk->ctrl_dma_rx) {
+		ib_dma_unmap_single(ibdev->dev, lnk->ctrl_dma_rx,
+				(sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+				    DMA_FROM_DEVICE);
+		lnk->ctrl_dma_rx = 0;
+	}
+	if (lnk->ctrl_dma_tx) {
+		ib_dma_unmap_single(ibdev->dev, lnk->ctrl_dma_tx,
+				(sizeof(struct rmb_e_ctrl) * lnk->send_wr_num),
+				    DMA_TO_DEVICE);
+		lnk->ctrl_dma_tx = 0;
+	}
+	if (lnk->roce_pd) {
+		ib_dealloc_pd(lnk->roce_pd);
+		lnk->roce_pd = NULL;
+	}
+}
+
+static int smc_pending_sends_on_link(struct smc_link *link)
+{
+	u32 i;
+
+	i = find_first_bit(link->pending_bit_mask, link->send_wr_num);
+	if (i < link->send_wr_num)
+		return 1;
+	return 0;
+}
+
+int smc_wait_no_pending_sends_on_link(struct smc_link *link)
+{
+	int rc = 1;
+
+	if (smc_pending_sends_on_link(link)) {
+		rc = wait_event_interruptible_timeout(link->wqe_waiter,
+			!smc_pending_sends_on_link(link),
+			SMC_WAIT_PENDING_SENDS_TIMEO);
+	}
+	return rc;
+}
+
+void smc_free_lgr(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry, *q;
+	struct rmb_tx_addrs *entrytx, *tmptx;
+	struct rmb_rx_addrs *entryrx, *tmprx;
+	int i, rc;
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_del_init(&lgr->list); /* remove from smc_lgr_freeing_list */
+	spin_unlock_bh(&smc_lgr_list.lock);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (smc_lnk_downing(&lgr->lnk[i].state))
+			rc = smc_wait_no_pending_sends_on_link(&lgr->lnk[i]);
+		smc_free_link(lgr, i);
+	}
+	cancel_delayed_work_sync(&lgr->free_lgr_work);
+	cancel_work_sync(&lgr->llc_add_link_work);
+	if (lgr->role == SMC_SERV)
+		cancel_work_sync(&lgr->llc_del_link_work);
+	cancel_work_sync(&lgr->llc_conf_rkey_work);
+	cancel_work_sync(&lgr->llc_del_rkey_work);
+	destroy_workqueue(lgr->llc_wq);
+	spin_lock_bh(&lgr->llc_event_q_lock);
+	list_for_each_entry_safe(qentry, q, &lgr->llc_event_q, list) {
+		list_del_init(&qentry->list);
+		kfree(qentry);
+	}
+	spin_unlock_bh(&lgr->llc_event_q_lock);
+	list_for_each_entry_safe(entrytx, tmptx, &lgr->rmb_tx_bufs[i], list) {
+		list_del(&entrytx->list);
+		kfree(entrytx->rmb_tx);
+		kfree(entrytx);
+	}
+	list_for_each_entry_safe(entryrx, tmprx, &lgr->rmb_rx_bufs[i], list) {
+		list_del(&entryrx->list);
+		kfree(entryrx->rmb_rx);
+		kfree(entryrx);
+	}
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		smc_free_link_mem(&lgr->lnk[i]);
+		lgr->lnk[i].lnk_grp = NULL;
+	}
+	kfree(lgr);
+}
+
+static void smc_free_lgr_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  free_lgr_work.work);
+	struct llc_qentry *qentry;
+	int i, rest_time, rc, peer_notified = 0, peer_gone;
+
+	if (list_empty(&lgr->list))
+		peer_gone = -EPIPE;
+	else
+		peer_gone = smc_check_peer_gone(lgr);
+	if ((lgr->role == SMC_CLNT) && !peer_gone)
+		return;		/* server to trigger lgr removal */
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	read_lock_bh(&lgr->conns_lock);
+	rc = RB_EMPTY_ROOT(&lgr->conns_all);
+	read_unlock_bh(&lgr->conns_lock);
+	if (!rc) {
+		spin_unlock_bh(&smc_lgr_list.lock);
+		return;
+	}
+	if (atomic_read(&smc_reconfiguring) ||
+	    mutex_is_locked(&lgr->conf_mutex)) {
+		/* postpone freeing */
+		spin_unlock_bh(&smc_lgr_list.lock);
+		schedule_delayed_work(&lgr->free_lgr_work,
+				      LLC_WAIT_FIRST_TIMEO);
+		return;
+	}
+	list_move_tail(&lgr->list, &smc_lgr_freeing_list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+	if (peer_gone)
+		goto out;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (!smc_lnk_downing(&lgr->lnk[i].state) ||
+		    (lgr->lgr_type == NONE) ||
+		    peer_notified)
+			continue;
+		rest_time = llc_initiate(lgr, LLC_GRP_DEL_LINK);
+		if (!rest_time) {		/* timeout */
+			llc_stop(lgr, LLC_GRP_DEL_LINK);
+			continue;
+		}
+		if (!&lgr->lnk[i].link_id) {	/* link already freed */
+			llc_stop(lgr, LLC_GRP_DEL_LINK);
+			continue;
+		}
+		atomic_set(&lgr->lnk[i].state, SMC_LINK_UP); /* temporarily */
+		rc = llc_send_del_link(&lgr->lnk[i], NULL, LLC_FLAG_DEL_ORDERLY,
+				       LLC_DEL_PROG_INIT_TERM, LLC_SERV);
+		if (rc >= 0) {
+			lgr->llc_ctl[LLC_SERV].qentry = NULL;
+			qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO,
+					  LLC_DEL_LINK);
+			if (qentry)
+				peer_notified = 1;
+		}
+		llc_stop(lgr, LLC_GRP_DEL_LINK);
+		atomic_set(&lgr->lnk[i].state, SMC_LINK_DOWN);
+	}
+out:
+	llc_kill_waiters(lgr);
+	cancel_work_sync(&lgr->llc_event_work);
+	schedule_delayed_work(&lgr->free_lgr_final_work, 2 * TCP_TIMEWAIT_LEN);
+}
+
+static void smc_free_lgr_final_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  free_lgr_final_work.work);
+
+	smc_free_lgr(lgr);
+}
+
+static inline int smc_splice_in_use(const struct smc_sock *smc)
+{
+	return smc->spd != NULL;
+}
+
+static int smc_splice_in_progress(struct smc_sock *smc)
+{
+	struct smc_curs curs;
+	struct smc_curs spl_curs;
+
+	spl_curs.s.lcurs = atomic64_read(&smc->conn.local_rx_spliced.s.acurs);
+	if (spl_curs.s.curs.reserved0) {
+		curs.s.lcurs =
+		       atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+		if (smc_curs_diff(smc->conn.rmb_rx_size, &curs, &spl_curs))
+			return 1;
+	}
+
+	return 0;
+}
+
+static inline int smc_find_pending_send(struct smc_link *link, u64 wr_id)
+{
+	u32 i;
+
+	for (i = 0; i < link->send_wr_num; i++) {
+		if (link->pending[i].wr_id_send == wr_id)
+			return i;
+	}
+	return link->send_wr_num;
+}
+
+static void smc_flag_sock_err(struct smc_sock *smc)
+{
+	int old_state;
+
+	sock_hold(&smc->sk);
+	smc->sk.sk_err = ECONNABORTED;
+	lock_sock(&smc->sk);
+	old_state = smc->sk.sk_state;
+	smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+	if (smc->tcpsocket && smc->tcpsocket->sk) {
+		smc->tcpsocket->sk->sk_err = ECONNABORTED;
+		smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+	}
+	smc_conn_term(smc);
+	if (!sock_flag(&smc->sk, SOCK_DEAD))
+		smc->sk.sk_error_report(&smc->sk);
+	release_sock(&smc->sk);
+	if (old_state != SMC_INIT) {
+		/* forget about data prepared to be sent */
+		atomic64_set(&smc->conn.tx_curs_prep.s.acurs,
+			     atomic64_read(&smc->conn.tx_curs_sent.s.acurs));
+		wake_up_interruptible(&smc->rx_waiter);
+		smc_sock_wake_tx(&smc->sk);
+		smc_sock_wake_rx(&smc->sk);
+		wake_up(&smc->prep_waiter);
+	}
+	sock_put(&smc->sk);
+	if ((old_state != SMC_INIT) &&
+	    (smc->sk.sk_state == SMC_CLOSED) &&
+	    (sock_flag(&smc->sk, SOCK_DEAD)) &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		schedule_work(&smc->release_work);
+	}
+}
+
+void smc_terminate_conn(struct smc_link_group *lgr)
+{
+	struct smc_sock *smc;
+	struct rb_node *node;
+
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_del_init(&lgr->list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+again:
+	read_lock_bh(&lgr->conns_lock);
+	for (node = rb_first(&lgr->conns_all); node; node = rb_next(node)) {
+		smc = container_of(rb_entry(node, struct smc_connection,
+					    alert_node),
+				   struct smc_sock, conn);
+		if ((smc->sk.sk_err != ECONNABORTED) && (!smc->use_tcp)) {
+			read_unlock_bh(&lgr->conns_lock);
+			smc_flag_sock_err(smc);
+			goto again;
+		}
+	}
+	read_unlock_bh(&lgr->conns_lock);
+}
+
+static int smc_write_space(struct smc_sock *);
+
+static int smc_switch_cursor(struct smc_sock *smc, int to_lnk_idx)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_curs cons, fin;
+	int diff;
+	int rc = 0;
+
+	atomic64_set(&conn->tx_curs_sent.s.acurs,
+		     atomic64_read(&conn->tx_curs_fin.s.acurs));
+	fin.s.lcurs = atomic64_read(&conn->local_tx_ctrl_fin.s.acurs);
+	atomic64_set(&conn->local_tx_ctrl.p_curs.s.acurs, fin.s.lcurs);
+	cons.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	if (((cons.s.curs.w > fin.s.curs.w) ||
+	     ((cons.s.curs.w == fin.s.curs.w) &&
+	      (cons.s.curs.c > fin.s.curs.c))) ||
+	    (!cons.s.curs.w && (fin.s.curs.w == 0xffff))) {
+		diff = smc_curs_diff(conn->peer_rx_buf_len, &fin, &cons);
+		smc_curs_add(conn->rmb_tx_size,
+			     &conn->tx_curs_sent.s.curs, diff);
+		smc_curs_add(conn->rmb_tx_size,
+			     &conn->tx_curs_fin.s.curs, diff);
+		smp_mb__before_atomic();
+		atomic_add(diff, &conn->tx_buf_space);
+		smp_mb__after_atomic();
+		smc_curs_add(conn->peer_rx_buf_len,
+			     &conn->local_tx_ctrl.p_curs.s.curs, diff);
+		smc_curs_add(conn->peer_rx_buf_len,
+			     &conn->local_tx_ctrl_fin.s.curs, diff);
+	}
+	atomic_set(&smc->conn.rx_buf_space, smc_write_space(smc));
+	if ((smc->sk.sk_state != SMC_INIT) &&
+	    (smc->sk.sk_state != SMC_CLOSED) &&
+	    (smc->sk.sk_state != SMC_DESTRUCT)) {
+		conn->local_tx_ctrl.p_flags.last_msg_valid_req = 1;
+		rc = smc_ctrl_send(conn, conn->lnk_idx);
+		conn->local_tx_ctrl.p_flags.last_msg_valid_req = 0;
+		schedule_delayed_work(&smc->write_work, 0);
+		smc_sock_wake_rx_for_tx(&smc->sk);
+	}
+	return rc;
+}
+
+int smc_switch_conns(struct smc_link_group *lgr, int lnk_idx, int dev_err)
+{
+	struct smc_sock *smc;
+	struct smc_connection *conn;
+	struct smc_roce_defs *rd_idx = &lgr->lnk[lnk_idx].roce_defs;
+	int to_lnk_idx, i, rc = 0, found = 0;
+	struct rb_node *node;
+
+	wake_up(&lgr->lnk[lnk_idx].wqe_waiter);
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		struct smc_roce_defs *rd_i = &lgr->lnk[i].roce_defs;
+
+		if ((atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP) ||
+		    (i == lnk_idx))
+			continue;
+		if (dev_err && (rd_i->ibdev == rd_idx->ibdev) &&
+		    (rd_i->port == rd_idx->port))
+			continue;
+		found = 1;
+		to_lnk_idx = i;
+		break;
+	}
+	if (!found) {
+		lgr->lgr_type = NONE;
+		wake_up_interruptible(&lgr->llc_waiter);
+		smc_terminate_conn(lgr); /* requires lgr_list_lock and
+					  * lgr->conns_lock
+					  */
+		return -ENOENT;
+	}
+
+again:
+	read_lock_bh(&lgr->conns_lock);
+	for (node = rb_first(&lgr->conns_all); node; node = rb_next(node)) {
+		conn = rb_entry(node, struct smc_connection, alert_node);
+		if (conn->lnk_idx != lnk_idx)
+			continue;
+		smc = container_of(conn, struct smc_sock, conn);
+		if ((smc->sk.sk_state == SMC_CLOSED) ||
+		    (smc->sk.sk_state == SMC_PEERCLW2) ||
+		    (smc->sk.sk_state == SMC_PEERFINCLW) ||
+		    (smc->sk.sk_state == SMC_APPLFINCLW) ||
+		    (smc->sk.sk_state == SMC_APPLCLW2) ||
+		    (smc->sk.sk_state == SMC_PROCESSABORT)) {
+			conn->lnk_idx = to_lnk_idx;
+			continue;
+		}
+		sock_hold(&smc->sk);
+		read_unlock_bh(&lgr->conns_lock);
+		lock_sock(&smc->sk);
+		conn->lnk_idx = to_lnk_idx;
+		if (!(atomic64_read(&conn->tx_curs_prep.s.acurs)) &&
+		    !(atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs)) &&
+		    !conn->local_tx_ctrl.conn_state_flags.closed_conn) {
+			release_sock(&smc->sk);
+			sock_put(&smc->sk);
+			goto again;
+		}
+		rc = smc_switch_cursor(smc, to_lnk_idx);
+		release_sock(&smc->sk);
+		sock_put(&smc->sk);
+		if (rc) {
+			smc_link_down(lgr, to_lnk_idx);
+			break;
+		}
+		goto again;
+	}
+	read_unlock_bh(&lgr->conns_lock);
+	return to_lnk_idx;
+}
+
+void smc_cq_event_handler(struct ib_event *evt, void *ptr)
+{
+	return;
+}
+
+static void smc_qp_event_handler(struct ib_event *evt, void *ptr)
+{
+	struct smc_link *link = (struct smc_link *)ptr;
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_roce_defs *rocdefs = &link->roce_defs;
+	struct smc_ib_device *smc_ibdev = (struct smc_ib_device *)evt->device;
+	int lnk_idx;
+
+	lnk_idx = link - &lgr->lnk[0];
+
+	switch (evt->event) {
+	case IB_EVENT_DEVICE_FATAL:	/* 8 */
+		if (cmpxchg(&smc_ibdev->port_err[rocdefs->port - 1], 0, 1) == 0)
+			schedule_work(&rocdefs->ibdev->port_err_work);
+		break;
+	case IB_EVENT_GID_CHANGE:	 /* 17 */
+	case IB_EVENT_PORT_ERR:		 /* 10 */
+	case IB_EVENT_QP_ACCESS_ERR:	 /* 3 */
+		if ((atomic_read(&link->state) == SMC_LINK_UP) &&
+		    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0))
+			schedule_work(&lgr->link_down_work);
+		break;
+	case IB_EVENT_COMM_EST:		 /* 4 */
+	case IB_EVENT_PORT_ACTIVE:	 /* 9 */
+	default:
+		break;
+	}
+}
+
+static inline void smc_send_tasklet_conn(struct smc_sock *smc)
+{
+	if (smc->sk.sk_socket && atomic_read(&smc->conn.tx_buf_space) &&
+	    test_bit(SOCK_NOSPACE, &smc->sk.sk_socket->flags)) {
+		smc_sock_wake_tx(&smc->sk);
+	}
+	if (smc->sk.sk_state != SMC_ACTIVE)
+		wake_up(&smc->destruct_waiter);
+}
+
+static inline void smc_send_tasklet_wc(struct ib_wc *wc)
+{
+	struct smc_link_group *lgr;
+	struct smc_link *link;
+	struct smc_sock *smc;
+	struct pending_send pnd_snd;
+	u32 pnd_snd_idx;
+	int i, diff;
+
+	link = wc->qp->qp_context;
+	if (!link->link_id ||
+	    (memcmp(link->lnk_eyecatch, SMC_LNK_EYE_CATCHER,
+		    SMC_LNK_EYE_CATCH_LEN)))
+		return;
+	pnd_snd_idx = smc_find_pending_send(link, wc->wr_id);
+	if (pnd_snd_idx == link->send_wr_num)
+		return;
+	memcpy(&pnd_snd, &link->pending[pnd_snd_idx],
+	       sizeof(struct pending_send));
+	if (!test_and_clear_bit(pnd_snd_idx, link->pending_bit_mask))
+		return;
+	lgr = link->lnk_grp;
+	if (wc->status) {
+		int lnk_idx = link - &lgr->lnk[0];
+
+		for_each_set_bit(i, link->pending_bit_mask, link->send_wr_num)
+			clear_bit(i, link->pending_bit_mask);
+		if ((wc->wr_id != 1) &&
+		    (cmpxchg(&lgr->lnk_down[lnk_idx], 0, 1) == 0))
+			schedule_work(&lgr->link_down_work);
+	}
+	if (pnd_snd.conn) {
+		smc = container_of(pnd_snd.conn, struct smc_sock, conn);
+		bh_lock_sock(&smc->sk);
+		if ((!wc->status) &&
+		    (link == &lgr->lnk[pnd_snd.conn->lnk_idx])) {
+			diff = smc_curs_diff(pnd_snd.conn->rmb_tx_size,
+					     &pnd_snd.conn->tx_curs_fin,
+					     &pnd_snd.cursor);
+			smp_mb__before_atomic();
+			atomic_add(diff, &pnd_snd.conn->tx_buf_space);
+			smp_mb__after_atomic();
+			atomic64_set(&pnd_snd.conn->tx_curs_fin.s.acurs,
+				     pnd_snd.cursor.s.lcurs);
+			atomic64_set(&pnd_snd.conn->local_tx_ctrl_fin.s.acurs,
+				     pnd_snd.p_cursor.s.lcurs);
+			pnd_snd.conn->tx_seq_fin = pnd_snd.ctrl_seq;
+		}
+		wake_up(&link->wqe_waiter);
+		smc_send_tasklet_conn(smc);
+		bh_unlock_sock(&smc->sk);
+	} else {
+		if (pnd_snd.post_ctl != LLC_NO_NOTIFY) {
+			lgr->llc_ctl[pnd_snd.post_ctl].ctrl_elem =
+							LLC_SEND_POSTED;
+			lgr->llc_ctl[pnd_snd.post_ctl].wc_status = wc->status;
+			wake_up_interruptible(&lgr->llc_waiter);
+		}
+		wake_up(&link->wqe_waiter);
+	}
+}
+
+void smc_send_tasklet_fn(unsigned long data)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)data;
+	struct ib_wc wc[SMC_MAX_POLL_CQE];
+	int i = 0, rc, rc1 = 0;
+	int polled = 0;
+
+again:
+	polled++;
+	do {
+		rc = ib_poll_cq(ibdev->roce_cq_send, SMC_MAX_POLL_CQE, wc);
+		if (polled == 1) {
+			rc1 = ib_req_notify_cq(ibdev->roce_cq_send,
+					       IB_CQ_NEXT_COMP |
+					       IB_CQ_REPORT_MISSED_EVENTS);
+		}
+		if (!rc)
+			break;
+		for (i = 0; i < rc; i++)
+			smc_send_tasklet_wc(&wc[i]);
+	} while (rc > 0);
+	if (polled == 1)
+		goto again;
+}
+
+void smc_cq_handler_send(struct ib_cq *ib_cq, void *cq_context)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)cq_context;
+
+	tasklet_schedule(&ibdev->send_tasklet);
+}
+
+static inline
+void smc_save_cursor(struct smc_curs *local, struct rmb_curs *peer)
+{
+	struct smc_curs temp, old;
+
+	old.s.lcurs = atomic64_read(&local->s.acurs);
+	temp.s.lcurs = atomic64_read(&peer->r.acurs);
+	temp.s.curs.c = ntohl(temp.s.curs.c);
+	temp.s.curs.w = ntohs(temp.s.curs.w);
+	if ((old.s.curs.w > temp.s.curs.w) && temp.s.curs.w)
+		return;
+	if ((old.s.curs.w == temp.s.curs.w) &&
+	    (old.s.curs.c > temp.s.curs.c))
+		return;
+	atomic64_set(&local->s.acurs, temp.s.lcurs);
+}
+
+static inline
+void smc_set_cursor(struct smc_curs *local, struct rmb_cursor *peer)
+{
+	struct smc_curs temp;
+
+	temp.s.lcurs = atomic64_read(&local->s.acurs);
+	peer->count = htonl(temp.s.curs.c);
+	peer->wwrap_seq = htons(temp.s.curs.w);
+}
+
+static inline
+void smc_save_ctrl(struct smc_e_ctrl *local, struct rmb_e_ctrl *peer)
+{
+	local->p_flags = peer->prod_flags;
+	local->conn_state_flags = peer->conn_state_flags;
+	smc_save_cursor(&local->p_curs, &peer->p_curs);
+	smc_save_cursor(&local->c_curs, &peer->c_curs);
+	local->ctrl_type = peer->ctrl_type;
+	local->ctrl_len = peer->ctrl_len;
+	local->ctrl_seq = ntohs(peer->ctrl_seq);
+	local->ctrl_token = peer->ctrl_token;
+}
+
+static inline
+void smc_set_ctrl(struct smc_e_ctrl *local, struct rmb_e_ctrl *peer)
+{
+	memcpy((char *)peer, (char *)local, SMC_SIZE_OF_CTRL_DATA);
+	peer->prod_flags = local->p_flags;
+	peer->conn_state_flags = local->conn_state_flags;
+	smc_set_cursor(&local->p_curs, &peer->p_curs.r.curs);
+	smc_set_cursor(&local->c_curs, &peer->c_curs.r.curs);
+	peer->ctrl_seq = htons(local->ctrl_seq);
+	peer->ctrl_token = local->ctrl_token;
+}
+
+static void smc_update_cons_curs(struct smc_sock *, struct smc_link *link);
+
+static int smc_recv_validate(struct smc_sock *smc, struct smc_link *link,
+			     u32 index)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+	u16 tmp;
+
+	/* check that seqnum was seen before */
+	tmp = conn->ctrl_seq_rcvd - ntohs(link->ctrl_buf_rx[index].ctrl_seq);
+	if (tmp > 0x7fff) {
+		/* drop connection */
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+		conn->local_rx_ctrl.conn_state_flags.abnormal_close = 1;
+		if (smc->tcpsocket && smc->tcpsocket->sk) {
+			smc->tcpsocket->sk->sk_err = ECONNABORTED;
+			smc->tcpsocket->sk->sk_data_ready(smc->tcpsocket->sk);
+		}
+		conn->lnk_idx = link - &conn->lgr->lnk[0];
+		smc_conn_term(smc);
+		wake_up_interruptible(&smc->rx_waiter);
+		smc_sock_wake_tx(&smc->sk);
+		smc_sock_wake_rx(&smc->sk);
+		if ((smc->sk.sk_state == SMC_CLOSED) &&
+		    (sock_flag(&smc->sk, SOCK_DEAD)) &&
+		    (cmpxchg(&smc->sock_put_done, 0, 1) == 0))
+			schedule_work(&smc->release_work);
+		rc = -ECONNABORTED;
+	} else {
+		conn->local_rx_ctrl.p_flags.cons_curs_upd_req = 1;
+	}
+	return rc;
+}
+
+static void smc_recv_action(struct smc_sock *smc,
+			    struct smc_link *link, u32 index)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_curs prod, cons_old, prod_old;
+	int diff;
+
+	if (link->ctrl_buf_rx[index].prod_flags.last_msg_valid_req) {
+		if (smc_recv_validate(smc, link, index))
+			return;
+	}
+	prod_old.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons_old.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	smc_save_ctrl(&conn->local_rx_ctrl, &link->ctrl_buf_rx[index]);
+	if (!conn->local_rx_ctrl.p_flags.last_msg_valid_req) {
+		if ((conn->local_rx_ctrl.ctrl_seq -
+			(u16)(conn->ctrl_seq_rcvd + 1)) > 0x7fff)
+			return;
+		conn->ctrl_seq_rcvd = conn->local_rx_ctrl.ctrl_seq;
+	}
+	diff = smc_curs_diff(conn->peer_rx_buf_len, &cons_old,
+			     &conn->local_rx_ctrl.c_curs);
+	smp_mb__before_atomic();
+	atomic_add(diff, &conn->rx_buf_space);
+	smp_mb__after_atomic();
+	diff = smc_curs_diff(conn->rmb_rx_size, &prod_old,
+			     &conn->local_rx_ctrl.p_curs);
+	smp_mb__before_atomic();
+	atomic_add(diff, &conn->bytes_to_rcv);
+	smp_mb__after_atomic();
+
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		smc->sk.sk_err = ECONNRESET;
+	if (smc_stop_received(conn)) {
+		smc->sk.sk_shutdown |= RCV_SHUTDOWN;
+		sock_set_flag(&smc->sk, SOCK_DONE);
+		smc_conn_term(smc);
+		smc_sock_wake_rx(&smc->sk);
+		smc_sock_wake_tx(&smc->sk);
+	}
+
+	/* piggy backed tx info */
+	smc_sock_wake_rx_for_tx(&smc->sk);
+
+	if ((smc->sk.sk_state == SMC_CLOSED) &&
+	    smc_close_received(conn) &&
+	    sock_flag(&smc->sk, SOCK_DEAD) &&
+	    !smc->sk.sk_socket &&
+	    (cmpxchg(&smc->sock_put_done, 0, 1) == 0)) {
+		wake_up_interruptible(&smc->rx_waiter);
+		/* socket is closed and peer has confirmed close, now we are
+		 * allowed to clean up our sock/rmbe
+		 */
+		schedule_work(&smc->release_work);
+		return;
+	}
+
+	/* socket connected but not accepted */
+	if (!smc->sk.sk_socket)
+		return;
+
+	/* data available */
+	if (conn->local_rx_ctrl.p_flags.urg_data_present) {
+		prod.s.lcurs =
+			atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+		smc_curs_dec(conn->rmb_rx_size, &prod.s.curs);
+		atomic64_set(&conn->rx_urg_curs.s.acurs, prod.s.lcurs);
+		/* set urg_data to previous byte */
+		conn->rx_urg_data =
+			conn->rmb_rx_elem->rmb_rx[prod.s.curs.c];
+		conn->rx_urg_state = SMC_URG_VALID;
+		sk_send_sigurg(&smc->sk);
+	}
+	if ((conn->local_rx_ctrl.p_flags.write_blocked) ||
+	    (conn->local_rx_ctrl.p_flags.cons_curs_upd_req))
+		smc_update_cons_curs(smc, link);
+	if ((smc_to_read(smc) > 0) ||
+	    conn->local_rx_ctrl.p_flags.urg_data_pending ||
+	    smc_stop_received(conn) ||
+	    smc->sk.sk_shutdown & RCV_SHUTDOWN)
+		smc_sock_wake_rx(&smc->sk);
+}
+
+static inline void smc_recv_conn_update(struct rmb_e_ctrl *ctrl,
+					struct smc_link *link, u64 wr_id)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_connection *connection;
+	struct smc_sock *smc;
+	u32 index;
+
+	/* lookup connection */
+	read_lock_bh(&lgr->conns_lock);
+	connection = smc_find_conn(ctrl->ctrl_token, lgr);
+	if (!connection) {
+		read_unlock_bh(&lgr->conns_lock);
+	} else {
+		smc = container_of(connection, struct smc_sock, conn);
+		if (smc->sk.sk_state != SMC_DESTRUCT) {
+			sock_hold(&smc->sk);
+			read_unlock_bh(&lgr->conns_lock);
+			index = wr_id % link->recv_wr_num;
+			bh_lock_sock(&smc->sk);
+			smc_recv_action(smc, link, index);
+			bh_unlock_sock(&smc->sk);
+			sock_put(&smc->sk); /* no free sk in softirq-context */
+		} else {
+			read_unlock_bh(&lgr->conns_lock);
+		}
+	}
+}
+
+static inline void smc_recv_process_wqs(struct ib_wc wc[], int num)
+{
+	struct smc_link *link;
+	struct smc_link_group *lgr;
+	struct rmb_e_ctrl *ctrl;
+	int i, idx;
+	u32 index;
+
+	for (i = 0; i < num; i++) {
+		link = wc[i].qp->qp_context;
+		if (memcmp(link->lnk_eyecatch, SMC_LNK_EYE_CATCHER,
+			   SMC_LNK_EYE_CATCH_LEN))
+			continue;
+		lgr = link->lnk_grp;
+		if ((atomic_read(&link->state) == SMC_LINK_DOWN) ||
+		    (atomic_read(&link->state) == SMC_LINK_FREED))
+			continue;
+		if (wc[i].status == IB_WC_SUCCESS) {
+			index = wc[i].wr_id % link->recv_wr_num;
+			ctrl = (struct rmb_e_ctrl *)&link->ctrl_buf_rx[index];
+			link->rcv_tstamp = (u32)jiffies;
+			if (ctrl->ctrl_type == RMBE_CTRL)
+				smc_recv_conn_update(ctrl, link, wc[i].wr_id);
+			else
+				llc_enqueue(link, (struct llc_msg *)ctrl);
+			smc_outstanding_post_recv(link);
+		} else {
+			/* handle status errors
+			 * terminate all connections of this lgr abnormally
+			 */
+			switch (wc[i].status) {
+			case IB_WC_RETRY_EXC_ERR:
+			case IB_WC_RNR_RETRY_EXC_ERR:
+			case IB_WC_WR_FLUSH_ERR:
+				idx = link - &lgr->lnk[0];
+				if ((atomic_read(&link->state) == SMC_LINK_UP)
+				 && (cmpxchg(&lgr->lnk_down[idx], 0, 1) == 0))
+					schedule_work(&lgr->link_down_work);
+				break;
+			default:
+				smc_outstanding_post_recv(link);
+				break;
+			}
+			continue;
+		}
+	}
+}
+
+void smc_recv_tasklet_fn(unsigned long data)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)data;
+	struct ib_wc wc[SMC_MAX_POLL_CQE];
+	int rc;
+	int polled = 0;
+
+again:
+	polled++;
+	do {
+		memset(&wc, 0, sizeof(wc));
+		rc = ib_poll_cq(ibdev->roce_cq_recv, SMC_MAX_POLL_CQE, wc);
+		if (polled == 1)
+			ib_req_notify_cq(ibdev->roce_cq_recv,
+					 IB_CQ_SOLICITED_MASK
+				       | IB_CQ_REPORT_MISSED_EVENTS);
+		if (!rc)
+			break;
+		smc_recv_process_wqs(&wc[0], rc);
+	} while (rc > 0);
+	if (polled == 1)
+		goto again;
+}
+
+void smc_cq_handler_recv(struct ib_cq *ib_cq, void *cq_context)
+{
+	struct smc_ib_device *ibdev = (struct smc_ib_device *)cq_context;
+
+	tasklet_schedule(&ibdev->recv_tasklet);
+}
+
+static int smc_alloc_link_mem(struct smc_link *link)
+{
+	/* allocate link related memory */
+	link->ctrl_buf_rx = (struct rmb_e_ctrl *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct rmb_e_ctrl),
+			GFP_KERNEL);
+	if (!link->ctrl_buf_rx)
+		goto no_mem;
+	link->ctrl_buf_tx = (struct rmb_e_ctrl *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct rmb_e_ctrl),
+			GFP_KERNEL);
+	if (!link->ctrl_buf_tx)
+		goto no_mem_ctrl_buf_rx;
+	link->send_wr = (struct ib_send_wr *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct ib_send_wr),
+			GFP_KERNEL);
+	if (!link->send_wr)
+		goto no_mem_ctrl_buf_tx;
+	link->recv_wr = (struct ib_recv_wr *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct ib_recv_wr),
+			GFP_KERNEL);
+	if (!link->recv_wr)
+		goto no_mem_send_wr;
+	link->send_sge = (struct ib_sge *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct ib_sge),
+			GFP_KERNEL);
+	if (!link->send_sge)
+		goto no_mem_recv_wr;
+	link->recv_sge = (struct ib_sge *)
+		kcalloc(smc_ctrl_buf_cnt * 10, sizeof(struct ib_sge),
+			GFP_KERNEL);
+	if (!link->recv_sge)
+		goto no_mem_send_sge;
+	link->pending_bit_mask = kzalloc(BITS_TO_LONGS(smc_ctrl_buf_cnt) *
+					 sizeof(unsigned long),
+					 GFP_KERNEL);
+	if (!link->pending_bit_mask)
+		goto no_mem_recv_sge;
+	link->pending = (struct pending_send *)
+		kcalloc(smc_ctrl_buf_cnt, sizeof(struct pending_send),
+			GFP_KERNEL);
+	if (!link->pending)
+		goto no_mem_pending_bit_mask;
+	return 0;
+
+no_mem_pending_bit_mask:
+	kfree(link->pending_bit_mask);
+no_mem_recv_sge:
+	kfree(link->recv_sge);
+no_mem_send_sge:
+	kfree(link->send_sge);
+no_mem_recv_wr:
+	kfree(link->recv_wr);
+no_mem_send_wr:
+	kfree(link->send_wr);
+no_mem_ctrl_buf_tx:
+	kfree(link->ctrl_buf_tx);
+no_mem_ctrl_buf_rx:
+	kfree(link->ctrl_buf_rx);
+no_mem:
+	return -ENOMEM;
+}
+
+static void smc_init_sge(struct smc_link *lnk)
+{
+	u32 i;
+
+	for (i = 0; i < lnk->send_wr_num; i++) {
+		lnk->send_wr[i].next = NULL;
+		lnk->send_wr[i].sg_list = &lnk->send_sge[i];
+		lnk->send_wr[i].num_sge = 1;
+		lnk->send_wr[i].opcode = IB_WR_SEND;
+		lnk->send_wr[i].send_flags = IB_SEND_SIGNALED
+					   | IB_SEND_SOLICITED | IB_SEND_INLINE;
+		lnk->send_sge[i].addr =
+				lnk->ctrl_dma_tx + i * SMC_SIZE_OF_CTRL_BUF;
+		lnk->send_sge[i].length = SMC_SIZE_OF_CTRL_DATA;
+		lnk->send_sge[i].lkey = lnk->mr_tx->lkey;
+	}
+	for (i = 0; i < lnk->recv_wr_num; i++) {
+		lnk->recv_wr[i].next = NULL;
+		lnk->recv_wr[i].sg_list = &lnk->recv_sge[i];
+		lnk->recv_wr[i].num_sge = 1;
+		lnk->recv_sge[i].addr =
+				lnk->ctrl_dma_rx + i * SMC_SIZE_OF_CTRL_BUF;
+		lnk->recv_sge[i].length = SMC_SIZE_OF_CTRL_BUF;
+		lnk->recv_sge[i].lkey = lnk->mr_tx->lkey;
+	}
+}
+
+int smc_create_link(struct smc_link_group *lgr, int lnk_idx,
+		    struct smc_roce_defs *rocdefs)
+{
+	struct smc_ib_device *smc_ibdev = rocdefs->ibdev;
+	struct ib_device *ibdev;
+	struct smc_link *lnk = &lgr->lnk[lnk_idx];
+	int i, rc = 0;
+
+	if (list_empty(&smc_ib_devices))
+		return -ENODEV;
+	atomic_set(&lnk->state, SMC_LINK_ACTIVATING);
+	init_waitqueue_head(&lnk->wqe_waiter);
+	ibdev = smc_ibdev->dev;
+	memcpy(&lnk->roce_defs, rocdefs, sizeof(struct smc_roce_defs));
+	lnk->path_mtu = smc_ibdev->pattr[rocdefs->port - 1].active_mtu;
+	while (1) {
+		lnk->link_id = atomic_inc_return(&lgr->link_num);
+		if (!lnk->link_id)
+			continue;
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			if ((i != lnk_idx) &&
+			    (lgr->lnk[i].link_id == lnk->link_id))
+				continue;
+		}
+		break;
+	}
+	/* create PD */
+	lnk->roce_pd = ib_alloc_pd(ibdev);
+	if (IS_ERR(lnk->roce_pd)) {
+		atomic_set(&lnk->state, SMC_LINK_FREED);
+		rc = PTR_ERR(lnk->roce_pd);
+		lnk->roce_pd = NULL;
+		return rc;
+	}
+	/* create QP */
+	{
+	struct ib_qp_init_attr qp_attr = {
+		.event_handler = smc_qp_event_handler,
+		.qp_context = lnk,
+		.send_cq = smc_ibdev->roce_cq_send,
+		.recv_cq = smc_ibdev->roce_cq_recv,
+		.srq = NULL,
+		.cap = {
+			.max_send_wr = 3 * smc_ctrl_buf_cnt,
+				/* include unsolicited rdma_writes as well,
+				 * otherwise mlx4 code reaches overflow
+				 */
+			.max_recv_wr = 10 * smc_ctrl_buf_cnt,
+			.max_send_sge = 4,
+			.max_recv_sge = 1,
+			.max_inline_data = SMC_SIZE_OF_CTRL_BUF,
+		},
+		.sq_sig_type = IB_SIGNAL_REQ_WR,
+		.qp_type = IB_QPT_RC,
+	};
+	lnk->roce_qp = ib_create_qp(lnk->roce_pd, &qp_attr);
+	if (IS_ERR(lnk->roce_qp)) {
+		rc = PTR_ERR(lnk->roce_qp);
+		lnk->roce_qp = NULL;
+		goto dealloc_pd;
+	}
+	}
+	smc_check_qp_attr(lnk);
+	lnk->wr_id_send = 0;
+	lnk->wr_id_recv = 0;
+	lnk->mr_tx = ib_get_dma_mr(lnk->roce_pd, IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(lnk->mr_tx)) {
+		rc = PTR_ERR(lnk->mr_tx);
+		lnk->mr_tx = NULL;
+		goto destroy_qp;
+	}
+	lnk->ctrl_dma_rx = ib_dma_map_single(ibdev, lnk->ctrl_buf_rx,
+				(sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+					     DMA_FROM_DEVICE);
+	if (ib_dma_mapping_error(ibdev, lnk->ctrl_dma_rx)) {
+		lnk->ctrl_dma_rx = 0;
+		rc = -EIO;
+		goto dereg_mr;
+	}
+	lnk->ctrl_dma_tx = ib_dma_map_single(ibdev, lnk->ctrl_buf_tx,
+				sizeof(struct rmb_e_ctrl) * lnk->send_wr_num,
+					     DMA_TO_DEVICE);
+	if (ib_dma_mapping_error(ibdev, lnk->ctrl_dma_tx)) {
+		rc = -EIO;
+		goto dma_unmap;
+	}
+	get_random_bytes_arch(((char *)&lnk->psn_initial) + 1, 3);
+	lnk->rcv_tstamp = 0;
+	lnk->probes_out = 0;
+	smc_init_sge(lnk);
+	memcpy(lnk->lnk_eyecatch, SMC_LNK_EYE_CATCHER, SMC_LNK_EYE_CATCH_LEN);
+	memset(lnk->pending_bit_mask, 0,
+	       BITS_TO_LONGS(smc_ctrl_buf_cnt) * sizeof(unsigned long));
+	return 0;
+
+dma_unmap:
+	ib_dma_unmap_single(ibdev, lnk->ctrl_dma_rx,
+			    (sizeof(struct rmb_e_ctrl) * lnk->recv_wr_num),
+			    DMA_FROM_DEVICE);
+	lnk->ctrl_dma_rx = 0;
+dereg_mr:
+	ib_dereg_mr(lnk->mr_tx);
+	lnk->mr_tx = NULL;
+destroy_qp:
+	ib_destroy_qp(lnk->roce_qp);
+	lnk->roce_qp = NULL;
+dealloc_pd:
+	ib_dealloc_pd(lnk->roce_pd);
+	lnk->roce_pd = NULL;
+	lnk->link_id = 0;
+	memset(&lnk->roce_defs, 0, sizeof(struct smc_roce_defs));
+	atomic_set(&lnk->state, SMC_LINK_FREED);
+	return rc;
+}
+
+static inline
+struct rmb_tx_addrs *smc_get_tx_rmb_slot(struct smc_link_group *lgr,
+					 int start)
+{
+	struct rmb_tx_addrs *slot;
+
+	list_for_each_entry(slot, &lgr->rmb_tx_bufs[start], list) {
+		if (cmpxchg(&slot->used, 0, 1) == 0)
+			return slot;
+	}
+	return NULL;
+}
+
+static inline
+struct rmb_rx_addrs *smc_get_rx_rmb_slot(struct smc_link_group *lgr,
+					 int start)
+{
+	struct rmb_rx_addrs *slot;
+
+	list_for_each_entry(slot, &lgr->rmb_rx_bufs[start], list) {
+		if (cmpxchg(&slot->used, 0, 1) == 0)
+			return slot;
+	}
+	return NULL;
+}
+
+int smc_map_rmbs_tx(struct smc_link_group *lgr, int lnk_idx, int tx_size,
+		    struct rmb_tx_addrs *elem)
+{
+	struct smc_ib_device *dev = lgr->lnk[lnk_idx].roce_defs.ibdev;
+	int rc = 0;
+
+	if (elem->rmb_tx_dma[lnk_idx])
+		return rc; /* already mapped */
+	elem->rmb_tx_dma[lnk_idx] =
+		(char *)ib_dma_map_single(dev->dev, elem->rmb_tx,
+					  tx_size, DMA_TO_DEVICE);
+	if (ib_dma_mapping_error(dev->dev,
+				 (dma_addr_t)elem->rmb_tx_dma[lnk_idx]))
+		rc = -EIO;
+	return rc;
+}
+
+int smc_map_rmbs_rx(struct smc_link_group *lgr, int lnk_idx, int rx_size,
+		    struct rmb_rx_addrs *elem)
+{
+	struct smc_ib_device *dev = lgr->lnk[lnk_idx].roce_defs.ibdev;
+	int rc = 0;
+
+	if (elem->rmb_rx_dma[lnk_idx])
+		return rc; /* already mapped */
+	elem->rmb_rx_dma[lnk_idx] =
+		(char *)ib_dma_map_single(dev->dev, elem->rmb_rx,
+					  rx_size, DMA_FROM_DEVICE);
+	if (ib_dma_mapping_error(dev->dev,
+				 (dma_addr_t)elem->rmb_rx_dma[lnk_idx]))
+		rc = -EIO;
+	return rc;
+}
+
+long smc_get_dma_mr(struct smc_link_group *lgr, int lnk_idx,
+		    struct rmb_rx_addrs *elem)
+{
+	struct ib_mr *mr;
+	long rc = 0;
+
+	if (elem->mr_rx[lnk_idx])
+		return 0; /* already done */
+	/* obtain unique key -
+	 * next invocation of ib_get_dma_mr returns a different key!
+	 */
+	mr = ib_get_dma_mr(lgr->lnk[lnk_idx].roce_pd,
+			   IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE);
+	if (!IS_ERR(mr)) {
+		elem->mr_rx[lnk_idx] = mr;
+	} else {
+		elem->mr_rx[lnk_idx] = NULL;
+		rc = PTR_ERR(mr);
+	}
+	return rc;
+}
+
+long smc_map_rmbs_to_link(struct smc_link_group *lgr, int lnk_idx,
+			  int reuse_lnk_idx)
+{
+	struct rmb_tx_addrs *tx_entry;
+	struct rmb_rx_addrs *rx_entry;
+	long rc = 0;
+	int i;
+
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(tx_entry, &lgr->rmb_tx_bufs[i], list) {
+			if (reuse_lnk_idx <= SMC_MAX_SYM_LINKS) {
+				tx_entry->rmb_tx_dma[lnk_idx] =
+					tx_entry->rmb_tx_dma[reuse_lnk_idx];
+			} else {
+				rc = smc_map_rmbs_tx(lgr, lnk_idx,
+					     smc_uncompress_bufsize(rol8(i, 4)),
+						     tx_entry);
+				if (rc)
+					goto out;
+			}
+		}
+		list_for_each_entry(rx_entry, &lgr->rmb_rx_bufs[i], list) {
+			if (reuse_lnk_idx <= SMC_MAX_SYM_LINKS) {
+				rx_entry->rmb_rx_dma[lnk_idx] =
+					rx_entry->rmb_rx_dma[reuse_lnk_idx];
+			} else {
+				rc = smc_map_rmbs_rx(lgr, lnk_idx,
+					     smc_uncompress_bufsize(rol8(i, 4)),
+						     rx_entry);
+				if (rc)
+					goto out;
+			}
+			rc = smc_get_dma_mr(lgr, lnk_idx, rx_entry);
+			if (rc)
+				goto out;
+		}
+	}
+	return rc;
+
+out:
+	smc_link_down(lgr, lnk_idx);
+	return rc;
+}
+
+long smc_map_rmbs(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	long rc = 0;
+	int i, j;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP)
+			continue;
+		for (j = 0; j < i; j++) {
+			if ((lgr->lnk[i].roce_defs.ibdev ==
+				lgr->lnk[j].roce_defs.ibdev) &&
+			    (lgr->lnk[i].roce_defs.port ==
+				lgr->lnk[j].roce_defs.port))
+				break;
+		}
+		if (!smc->conn.tx_rmb_reused) {
+			if (j == i) {
+				rc = smc_map_rmbs_tx(lgr, i,
+						     smc->conn.rmb_tx_size,
+						     smc->conn.rmb_tx_elem);
+				if (rc) {
+					smc_link_down(lgr, i);
+					continue;
+				}
+			} else {
+				smc->conn.rmb_tx_elem->rmb_tx_dma[i] =
+					smc->conn.rmb_tx_elem->rmb_tx_dma[j];
+			}
+		}
+		if (!smc->conn.rx_rmb_reused) {
+			if (j == i) {
+				rc = smc_map_rmbs_rx(lgr, i,
+						     smc->conn.rmb_rx_size,
+						     smc->conn.rmb_rx_elem);
+				if (rc) {
+					smc_link_down(lgr, i);
+					continue;
+				}
+			} else {
+				smc->conn.rmb_rx_elem->rmb_rx_dma[i] =
+					smc->conn.rmb_rx_elem->rmb_rx_dma[j];
+			}
+			rc = smc_get_dma_mr(lgr, i, smc->conn.rmb_rx_elem);
+			if (rc) {
+				smc_link_down(lgr, i);
+				continue;
+			}
+		}
+	}
+	return rc;
+}
+
+int smc_create_rmbs(struct smc_sock *smc)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_link_group *lgr = conn->lgr;
+	struct rmb_tx_addrs *tx_rmb = NULL;
+	struct rmb_rx_addrs *rx_rmb = NULL;
+	int rmem, wmem, tmp_sl, tmp_ss, sl, ss;
+
+	rmem = smc->sk.sk_rcvbuf;
+	wmem = smc->sk.sk_sndbuf;
+	ss = ror8(smc_compress_bufsize(wmem), 4);
+	sl = smc_uncompress_bufsize(rol8(ss, 4));
+	for (tmp_sl = sl, tmp_ss = ss;
+	     tmp_sl >= SMC_MIN_RMBE_SIZE && tmp_ss >= 0;
+	     tmp_sl = tmp_sl >> 1, tmp_ss--) {
+		tx_rmb = smc_get_tx_rmb_slot(lgr, tmp_ss);
+		if (tx_rmb) {
+			conn->tx_rmb_reused = 1;
+		} else {
+			tx_rmb = (struct rmb_tx_addrs *)
+				kzalloc(sizeof(struct rmb_tx_addrs),
+					GFP_KERNEL);
+			if (!tx_rmb)
+				return -ENOMEM;
+			tx_rmb->rmb_tx = kzalloc(tmp_sl,
+						 GFP_KERNEL | __GFP_NOWARN |
+						 __GFP_NOMEMALLOC |
+						 __GFP_NORETRY);
+			tx_rmb->used = 1;
+			if (tx_rmb->rmb_tx) {
+				list_add(&tx_rmb->list,
+					 &lgr->rmb_tx_bufs[tmp_ss]);
+			}
+		}
+		if (tx_rmb->rmb_tx) {
+			conn->rmb_tx_elem = tx_rmb;
+			memset(conn->rmb_tx_elem->rmb_tx, 0, tmp_sl);
+			break;
+		}
+	}
+	if (!conn->rmb_tx_elem) {
+		kfree(tx_rmb);
+		return -ENOMEM;
+	}
+	conn->rmb_tx_size = tmp_sl;
+	conn->rmb_tx_size_short = rol8(tmp_ss, 4);
+	conn->upd_cons_limit =
+		min_t(int, conn->rmb_rx_size / 10, SOCK_MIN_SNDBUF / 2);
+
+	ss = ror8(smc_compress_bufsize(rmem), 4);
+	sl = smc_uncompress_bufsize(rol8(ss, 4));
+	for (tmp_sl = sl, tmp_ss = ss;
+	     tmp_sl >= SMC_MIN_RMBE_SIZE && tmp_ss >= 0;
+	     tmp_sl = tmp_sl >> 1, tmp_ss--) {
+		rx_rmb = smc_get_rx_rmb_slot(lgr, tmp_ss);
+		if (rx_rmb) {
+			conn->rx_rmb_reused = 1;
+		} else {
+			rx_rmb = (struct rmb_rx_addrs *)
+				kzalloc(sizeof(struct rmb_rx_addrs),
+					GFP_KERNEL);
+			if (!rx_rmb)
+				break;
+			rx_rmb->rmb_rx = kzalloc(tmp_sl,
+						 GFP_KERNEL | __GFP_NOWARN |
+						 __GFP_NOMEMALLOC);
+			rx_rmb->used = 1;
+			if (rx_rmb->rmb_rx)
+				list_add(&rx_rmb->list,
+					 &lgr->rmb_rx_bufs[tmp_ss]);
+		}
+		if (rx_rmb && rx_rmb->rmb_rx) {
+			conn->rmb_rx_elem = rx_rmb;
+			memset(conn->rmb_rx_elem->rmb_rx, 0, tmp_sl);
+			break;
+		}
+	}
+	if (!conn->rmb_rx_elem) {
+		cmpxchg(&tx_rmb->used, 1, 0);
+		conn->rmb_tx_size = 0;
+		conn->rmb_tx_size_short = 0;
+		kfree(rx_rmb);
+		return -ENOMEM;
+	}
+	conn->rmb_rx_size = tmp_sl;
+	conn->rmb_rx_size_short = rol8(tmp_ss, 4);
+
+	return 0;
+}
+
+static struct smc_link *smc_link_match(struct smc_link_group *lgr,
+				       struct smc_clc_msg_hdr2 *hdr2,
+				       u32 qp_num, enum smc_role role,
+				       unsigned long *flags)
+{
+	int i;
+
+	if (mutex_is_locked(&lgr->conf_mutex)) {
+		write_unlock_bh(&lgr->conns_lock);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		mutex_lock(&lgr->conf_mutex);
+		mutex_unlock(&lgr->conf_mutex);
+		spin_lock_bh(&smc_lgr_list.lock);
+		write_lock_bh(&lgr->conns_lock);
+		return (struct smc_link *)LGR_RECONFIGURING;
+	}
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) &&
+		    !memcmp(lgr->lnk[i].gid_peer, &hdr2->gid, 16) &&
+		    !memcmp(lgr->lnk[i].mac_peer, hdr2->mac, 6) &&
+		    ((role == SMC_SERV) ||
+		     (lgr->lnk[i].qp_num_peer == qp_num)))
+			return &lgr->lnk[i];
+	}
+	return NULL;
+}
+
+static int smc_find_link(struct smc_link_group *lgr, int alt)
+{
+	int first = -ENOENT;
+	int second = -ENOENT;
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			first = i;
+			break;
+		}
+	}
+	if (!alt)
+		return first;
+	second = first;
+	for (i = ++first; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) {
+			second = i;
+			break;
+		}
+	}
+	return second;
+}
+
+static void smc_link_up_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr
+		= container_of(work, struct smc_link_group, link_up_work);
+
+	if (lgr->lnk_up) {
+		mutex_lock(&lgr->conf_mutex);
+		lgr->lnk_up = 0;
+		smc_link_up(lgr);
+		mutex_unlock(&lgr->conf_mutex);
+	}
+}
+
+static void smc_link_down_worker(struct work_struct *work)
+{
+	int i;
+
+	struct smc_link_group *lgr
+		= container_of(work, struct smc_link_group, link_down_work);
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk_down[i]) {
+			lgr->lnk_down[i] = 0;
+			if (smc_lnk_downing(&lgr->lnk[i].state)) {
+				llc_cancel_wait(&lgr->lnk[i]);
+				mutex_lock(&lgr->conf_mutex);
+				smc_link_down(lgr, i);
+				mutex_unlock(&lgr->conf_mutex);
+			}
+		}
+	}
+}
+
+static int smc_create_lgr(struct smc_sock *smc, __be32 target,
+			  enum smc_role role, struct smc_clc_msg_hdr2 *hdr2)
+{
+	struct smc_link_group *lgr;
+	int i, j, rc;
+	char wq_name[11];
+
+	/* create a new link group */
+	lgr = kzalloc(sizeof(*lgr), GFP_KERNEL);
+	if (!lgr)
+		return -ENOMEM;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		rc = smc_alloc_link_mem(&lgr->lnk[i]);
+		if (rc) {
+			for (j = 0; j < i; j++)
+				smc_free_link_mem(&lgr->lnk[j]);
+			kfree(lgr);
+			return -ENOMEM;
+		}
+		lgr->lnk[i].lnk_grp = lgr;
+	}
+	for (i = 0; i < 16; i++) {
+		INIT_LIST_HEAD(&lgr->rmb_tx_bufs[i]);
+		INIT_LIST_HEAD(&lgr->rmb_rx_bufs[i]);
+	}
+	memcpy(lgr->peer_peer_id, hdr2->peer_id, 8);
+	lgr->daddr = target;
+	lgr->role = role;
+	lgr->lgr_type = INIT;
+	spin_lock_init(&lgr->llc_ctl_lock);
+	INIT_LIST_HEAD(&lgr->llc_event_q);
+	spin_lock_init(&lgr->llc_event_q_lock);
+	lgr->delayed_q = NULL;
+	init_waitqueue_head(&lgr->llc_waiter);
+	INIT_WORK(&lgr->llc_event_work, llc_event_worker);
+	INIT_DELAYED_WORK(&lgr->free_lgr_work, smc_free_lgr_worker);
+	INIT_DELAYED_WORK(&lgr->free_lgr_final_work, smc_free_lgr_final_worker);
+	INIT_WORK(&lgr->llc_add_link_work, llc_process_add_link);
+	INIT_WORK(&lgr->llc_del_link_work, llc_process_del_link);
+	INIT_WORK(&lgr->llc_conf_rkey_work, llc_process_confirm_rkey);
+	INIT_WORK(&lgr->llc_del_rkey_work, llc_process_delete_rkey);
+	lgr->conns_all = RB_ROOT;
+	rwlock_init(&lgr->conns_lock);
+	lgr->lgr_id = atomic_add_return(256, &smc_lgr_num);
+	smc->conn.lgr = lgr;
+	write_lock_bh(&lgr->conns_lock);
+	smc_register_in_lgr(&smc->conn);
+	write_unlock_bh(&lgr->conns_lock);
+	mutex_init(&lgr->conf_mutex);
+	spin_lock_bh(&smc_lgr_list.lock);
+	list_add_tail(&lgr->list, &smc_lgr_list.list);
+	spin_unlock_bh(&smc_lgr_list.lock);
+	mutex_lock(&lgr->conf_mutex);
+	smc->conn.lnk_idx = 0;
+	snprintf(wq_name, sizeof(wq_name), "smcwq%u", lgr->lgr_id);
+	lgr->llc_wq = create_singlethread_workqueue(wq_name);
+	INIT_WORK(&lgr->link_up_work, smc_link_up_worker);
+	INIT_WORK(&lgr->link_down_work, smc_link_down_worker);
+	return 0;
+}
+
+static int smc_check_lgr_consistency(struct smc_sock *smc)
+{
+	u32 reason;
+	u8 flag;
+
+	if (!smc->conn.lgr && !smc->new_lgr) {
+		/* send decline with flag */
+		reason = SMC_CLC_DEC_SU; /* synchronization error */
+		flag = SMC_LINK_GROUP_OUT_OF_SYNCH;
+		clc_send_decline(smc, reason, flag);
+		smc->sk.sk_err = ENOLINK;
+		return -ENOLINK;
+	}
+	if (smc->conn.lgr && smc->new_lgr) {
+		/* something is wrong with this link group */
+		spin_lock_bh(&smc_lgr_list.lock);
+		list_del_init(&smc->conn.lgr->list);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		smc_unregister_in_lgr(&smc->conn); /* takes conns_lock */
+		/* send decline with flag */
+		reason = SMC_CLC_DEC_SU; /* synchronization error */
+		flag = SMC_LINK_GROUP_OUT_OF_SYNCH;
+		clc_send_decline(smc, reason, flag);
+		/* terminate connections */
+		smc->sk.sk_err = ENOLINK;
+		smc_terminate_conn(smc->conn.lgr);
+		return -ENOLINK;
+	}
+	return 0;
+}
+
+int smc_create_conn(struct smc_sock *smc, __be32 target, enum smc_role role,
+		    struct smc_roce_defs *rocdefs,
+		    struct smc_clc_msg_hdr2 *hdr2,
+		    u32 arrival_qp_num)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct smc_link_group *lgr;
+	struct smc_link *link;
+	unsigned long flags;
+	int alt = 0, rc;
+
+	while (atomic_read(&smc_reconfiguring) ||
+	       (cmpxchg(&smc_lgr_list.create_conn_pending, 0, 1) != 0))
+		msleep(20);
+	spin_lock_bh(&smc_lgr_list.lock);
+again:
+	list_for_each_entry(lgr, &smc_lgr_list.list, list) {
+		write_lock_bh(&lgr->conns_lock);
+		if ((!memcmp(lgr->peer_peer_id, hdr2->peer_id, 8)) &&
+		    (lgr->role == role) &&
+		    (lgr->vlan == rocdefs->vlan) &&
+		    ((role == SMC_CLNT) ||
+		     (lgr->conns_num < smc_max_conn_per_lgr))) {
+			link = smc_link_match(lgr, hdr2, arrival_qp_num, role,
+					      &flags);
+			if (!link) {
+				write_unlock_bh(&lgr->conns_lock);
+				continue;
+			}
+			if (link == (struct smc_link *)LGR_RECONFIGURING) {
+				write_unlock_bh(&lgr->conns_lock);
+				goto again;
+			}
+			conn->lgr = lgr;
+			smc_lgr_list.create_conn_pending = 0;
+			if (role == SMC_SERV) {
+				if (lgr->lgr_type == SYMMETRIC)
+					/* try load balancing */
+					alt = lgr->conns_num % 2;
+				conn->lnk_idx = smc_find_link(lgr, alt);
+			} else {
+				conn->lnk_idx = link - &lgr->lnk[0];
+			}
+			smc_register_in_lgr(conn);
+			write_unlock_bh(&lgr->conns_lock);
+			break;
+		}
+		write_unlock_bh(&lgr->conns_lock);
+	}
+	spin_unlock_bh(&smc_lgr_list.lock);
+	if (role == SMC_CLNT) {
+		rc = smc_check_lgr_consistency(smc);
+		if (rc) {
+			smc_lgr_list.create_conn_pending = 0;
+			return -ENOLINK;
+		}
+	}
+	if (!conn->lgr) {
+		smc->new_lgr = 1;
+		rc = smc_create_lgr(smc, target, role, hdr2);
+		smc_lgr_list.create_conn_pending = 0;
+		if (rc)
+			return rc;
+		lgr = conn->lgr;
+		rc = smc_create_link(lgr, 0, rocdefs);
+		if (rc) {
+			lgr->lgr_type = NONE;
+			mutex_unlock(&lgr->conf_mutex);
+			spin_lock_bh(&smc_lgr_list.lock);
+			list_del(&lgr->list);
+			spin_unlock_bh(&smc_lgr_list.lock);
+			kfree(lgr);
+			return rc;
+		}
+		link = &lgr->lnk[conn->lnk_idx];
+	} else {
+		mutex_lock(&lgr->conf_mutex);
+		smc->new_lgr = 0;
+	}
+	conn->local_tx_ctrl.ctrl_type = RMBE_CTRL;
+	conn->local_tx_ctrl.ctrl_len = SMC_SIZE_OF_CTRL_DATA;
+	conn->tx_seq = 0;
+	spin_lock_init(&conn->send_lock);
+	atomic_set(&conn->bytes_to_rcv, 0);
+	conn->lgr->vlan = rocdefs->vlan;
+	init_waitqueue_head(&smc->rx_waiter);
+	return 0;
+}
+
+u8 smc_compress_bufsize(int n)
+{
+	u8 compressed = 0;
+
+	n = (n - 1) >> 14;
+	compressed = ilog2(n) + 1;
+	if (compressed > SMC_RMBE_SIZE_VALUES)
+		compressed = SMC_RMBE_SIZE_VALUES;
+	compressed = rol8(compressed, 4);
+	return compressed;
+}
+
+int smc_uncompress_bufsize(u8 x)
+{
+	u8 left_bits = ror8(x, 4);
+	u32 size;
+
+	size = 0x00000001 << (((int)left_bits) + 14);
+	return size;
+}
+
+static int smc_rdma_write(struct smc_connection *conn, int target_offset,
+			  int lnk_idx, int num, struct tx_sge sge[])
+{
+	struct smc_link_group *lgr = conn->lgr;
+	struct smc_link *link;
+	struct ib_sge sge_send[4];
+	struct ib_send_wr send_wr;
+	struct ib_send_wr *failed_wr;
+	int i, j, rc;
+	int to = target_offset;
+
+	memset(&send_wr, 0, sizeof(send_wr));
+	memset(&sge_send, 0, 4 * sizeof(struct ib_sge));
+	link = &lgr->lnk[lnk_idx];
+	for (i = 0, j = 0; (i < num) && (j < 4); i++, j++) {
+		if ((to % lgr->cache_align_size) &&
+		    ((to / lgr->cache_align_size)) !=
+		     ((to + sge[i].len - 1) / lgr->cache_align_size)) {
+			int initial = lgr->cache_align_size -
+				      to % lgr->cache_align_size;
+
+			/* split into aligned and unaligned part */
+			sge_send[j].addr =
+				((u64)conn->rmb_tx_elem->rmb_tx_dma[lnk_idx]) +
+				sge[i].offset;
+			sge_send[j].length = initial;
+			sge_send[j].lkey = link->mr_tx->lkey;
+			to += initial;
+			sge[i].len -= initial;
+			sge[i].offset += initial;
+			j++;
+		}
+		sge_send[j].addr = ((u64)conn->rmb_tx_elem->rmb_tx_dma[lnk_idx])
+				 + sge[i].offset;
+		sge_send[j].length = sge[i].len;
+		sge_send[j].lkey = link->mr_tx->lkey;
+	}
+	send_wr.wr_id =
+		(u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	send_wr.opcode = IB_WR_RDMA_WRITE;
+	send_wr.wr.rdma.remote_addr =
+		lgr->rtok[conn->rtok_idx].vaddr[lnk_idx]
+	      + target_offset
+	      + ((conn->peer_tcp_conn_idx - 1) * (conn->peer_rx_buf_len));
+	send_wr.wr.rdma.rkey = lgr->rtok[conn->rtok_idx].rkey[lnk_idx];
+	send_wr.num_sge = j;
+	send_wr.sg_list = sge_send;
+	failed_wr = &send_wr;
+	rc = ib_post_send(link->roce_qp, &send_wr, &failed_wr);
+	if (rc) {
+		struct smc_roce_defs *rocd;
+
+		rocd = &link->roce_defs;
+		conn->local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+			if (cmpxchg(&rocd->ibdev->port_err[rocd->port - 1],
+				    0, 1) == 0)
+				schedule_work(&rocd->ibdev->port_err_work);
+	}
+	return rc;
+}
+
+static inline int smc_get_free_ctrl_buf(struct smc_link *link, u32 *elem)
+{
+	*elem = link->send_wr_num;
+	if ((atomic_read(&link->state) == SMC_LINK_DOWN) ||
+	    (atomic_read(&link->state) == SMC_LINK_FREED)) {
+		return -EPIPE;
+	}
+	for_each_clear_bit(*elem, link->pending_bit_mask, link->send_wr_num) {
+		if (!test_and_set_bit(*elem, link->pending_bit_mask))
+			return 0;
+	}
+	*elem = link->send_wr_num;
+	return -EBUSY;
+}
+
+int smc_get_ctrl_buf(struct smc_link *link, u32 *elem)
+{
+	int rc = 0;
+
+	if (in_softirq()) {
+		rc = smc_get_free_ctrl_buf(link, elem);
+		if (rc)
+			return rc;
+	} else {
+		rc = wait_event_interruptible_timeout(link->wqe_waiter,
+				(smc_get_free_ctrl_buf(link, elem) != -EBUSY),
+				SMC_WAIT_FREE_CTRL_BUF_TIMEO);
+		if (!rc)
+			return -EPIPE;
+		if (rc == -ERESTARTSYS)
+			return -EINTR;
+		if (*elem == link->send_wr_num)
+			return -EPIPE;
+	}
+	return 0;
+}
+
+static inline int smc_add_pending_send(struct smc_link *link,
+				       struct smc_connection *conn, u32 *elem)
+{
+	u64 wr_id;
+	int rc = 0;
+
+	wr_id = (u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	rc = smc_get_ctrl_buf(link, elem);
+	if (rc < 0)
+		return rc;
+	memset(&link->pending[*elem], 0, sizeof(struct pending_send));
+	link->pending[*elem].conn = conn;
+	link->pending[*elem].wr_id_send = wr_id;
+	link->send_wr[*elem].wr_id = wr_id;
+	return 0;
+}
+
+int smc_ctrl_send(struct smc_connection *conn, int lnk_idx)
+{
+	struct smc_link *link;
+	struct rmb_e_ctrl ctrl;
+	int rc = 0;
+	u32 elem;
+
+	link = &conn->lgr->lnk[lnk_idx];
+	rc = smc_add_pending_send(link, conn, &elem);
+	if (rc)
+		return rc;
+	spin_lock_bh(&conn->send_lock);
+	if (conn->local_tx_ctrl.p_flags.last_msg_valid_req) {
+		conn->local_tx_ctrl.ctrl_seq = conn->tx_seq_fin;
+	} else {
+		conn->tx_seq++;
+		conn->local_tx_ctrl.ctrl_seq = conn->tx_seq;
+	}
+	link->pending[elem].cursor.s.curs = conn->tx_curs_sent.s.curs;
+	link->pending[elem].p_cursor.s.curs =
+		conn->local_tx_ctrl.p_curs.s.curs;
+	link->pending[elem].ctrl_seq = conn->tx_seq;
+	smc_set_ctrl(&conn->local_tx_ctrl, &ctrl);
+	rc = smc_wr_send(link, conn, (u64)&ctrl, elem);
+	if (!rc) {
+		atomic64_set(&conn->rx_curs_confirmed.s.acurs,
+			    atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs));
+	} else {
+		test_and_clear_bit(elem, link->pending_bit_mask);
+	}
+	spin_unlock_bh(&conn->send_lock);
+	return rc;
+}
+
+/* Send wr via ib_post_send.
+ * Requires conn->send_lock being held if entered with an smc_connection
+ */
+int smc_wr_send(struct smc_link *link, struct smc_connection *conn, u64 addr,
+		u32 index)
+{
+	struct ib_send_wr *failed_wr;
+	struct smc_roce_defs *rocdefs = &link->roce_defs;
+	struct smc_ib_device *ibdev = rocdefs->ibdev;
+	int rc;
+
+	rc = ib_req_notify_cq(ibdev->roce_cq_send,
+			      IB_CQ_SOLICITED_MASK |
+			      IB_CQ_REPORT_MISSED_EVENTS);
+	memcpy(&link->ctrl_buf_tx[index], (void *)addr, SMC_SIZE_OF_CTRL_DATA);
+	failed_wr = &link->send_wr[index];
+
+	rc = ib_post_send(link->roce_qp, &link->send_wr[index], &failed_wr);
+	if (rc) {
+		if ((atomic_read(&link->state) != SMC_LINK_FREED) &&
+		    (cmpxchg(&ibdev->port_err[rocdefs->port - 1], 0, 1) == 0))
+				/* try failover to other link */
+			schedule_work(&ibdev->port_err_work);
+		if (conn) {
+			if (!conn->local_rx_ctrl.p_flags.write_blocked &&
+			    !conn->local_rx_ctrl.p_flags.cons_curs_upd_req) {
+				/* not invoked in softirq context */
+				spin_unlock_bh(&conn->send_lock);
+				msleep(100); /* delay to enable port_err_work
+					      * to be done
+					      */
+				spin_lock_bh(&conn->send_lock);
+			}
+		} else {
+			msleep(100);
+		}
+		rc = 0;
+	}
+	return rc;
+}
+
+static int smc_write_space(struct smc_sock *smc)
+{
+	struct smc_curs prod;
+	struct smc_curs cons;
+	int buffer_len = smc->conn.peer_rx_buf_len;
+	int space;
+
+	prod.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&smc->conn.local_rx_ctrl.c_curs.s.acurs);
+
+	/* determine rx_buf space */
+	space = buffer_len - smc_curs_diff(buffer_len, &cons, &prod);
+
+	return space;
+}
+
+static int smc_wait_rx_space(struct smc_sock *smc)
+{
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int space, rc = 0;
+
+	space = atomic_read(&conn->rx_buf_space);
+	if (space)
+		goto out;
+
+	release_sock(sk);
+	rc = wait_event_interruptible_timeout(smc->rx_waiter,
+		atomic_read(&conn->rx_buf_space) ||
+		sk->sk_err ||
+		smc_stop_received(conn),
+		SMC_WAIT_RX_SPACE_TIMEO);
+	if ((rc == -ERESTARTSYS) || !rc) {
+		/* interrupted or timeout */
+		smc_flag_sock_err(smc);
+		space = sock_error(sk);
+		lock_sock(sk);
+		goto out;
+	}
+	lock_sock(sk);
+	space = atomic_read(&conn->rx_buf_space);
+out:
+	return space;
+}
+
+static int smc_wait_tx_space(struct smc_sock *smc, int flags)
+{   /* derived from sk_stream_wait_memory */
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT);
+	int rc = 0;
+	DEFINE_WAIT(wait);
+
+	if (timeo == MAX_SCHEDULE_TIMEOUT)
+		timeo = (prandom_u32() % (HZ / 5)) + 2;
+	set_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+	if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN) ||
+	    conn->local_tx_ctrl.conn_state_flags.sending_done) {
+		rc = -EPIPE;
+		goto out;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close) {
+		rc = -ECONNRESET;
+		goto out;
+	}
+	if (!timeo) {
+		rc = -EAGAIN;
+		goto out;
+	}
+	if (signal_pending(current)) {
+		rc = -EINTR;
+		goto out;
+	}
+	clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
+	if (atomic_read(&conn->tx_buf_space))
+		goto out;
+	set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+	rc = sk_wait_event(sk, &timeo, sk->sk_err ||
+			   (sk->sk_shutdown & SEND_SHUTDOWN) ||
+			   conn->local_tx_ctrl.conn_state_flags.sending_done ||
+			   smc_close_received(conn) ||
+			   atomic_read(&conn->tx_buf_space));
+	clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+	if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN) ||
+	    conn->local_tx_ctrl.conn_state_flags.sending_done) {
+		rc = -EPIPE;
+	}
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		rc = -ECONNRESET;
+out:
+	finish_wait(sk_sleep(sk), &wait);
+	return rc;
+}
+
+int smc_prepared_sends(struct smc_connection *conn)
+{
+	struct smc_curs prep;
+	struct smc_curs sent;
+
+	prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	sent.s.lcurs = atomic64_read(&conn->tx_curs_sent.s.acurs);
+	return smc_curs_diff(conn->rmb_tx_size, &sent, &prep);
+}
+
+int smc_pending_sends(struct smc_connection *conn)
+{
+	struct smc_link *lnk;
+	u32 i, j;
+
+	if ((!conn) || (!conn->lgr))
+		return 0;
+
+	for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+		lnk = &conn->lgr->lnk[j];
+		for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+			if (lnk->pending[i].conn == conn)
+				return 1;
+		}
+	}
+	return 0;
+}
+
+void smc_clear_pending_sends(struct smc_connection *conn)
+{
+	struct smc_link *lnk;
+	u32 i, j;
+
+	for (j = 0; j <= SMC_MAX_SYM_LINKS; j++) {
+		lnk = &conn->lgr->lnk[j];
+		for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+			if (lnk->pending[i].conn == conn)
+				clear_bit(i, lnk->pending_bit_mask);
+		}
+	}
+}
+
+static inline int smc_give_up_send(struct smc_sock *smc, int copied)
+{
+	struct smc_connection *conn = &smc->conn;
+
+	if (smc->sk.sk_shutdown & SEND_SHUTDOWN ||
+	    conn->local_tx_ctrl.conn_state_flags.abnormal_close)
+		return -EPIPE;
+	if (conn->local_rx_ctrl.conn_state_flags.abnormal_close)
+		return copied ? copied : -ECONNRESET;
+	return 0;
+}
+
+int smc_conn_send(struct smc_sock *smc, struct msghdr *msg, size_t len)
+{
+	struct smc_connection *conn = &smc->conn;
+	struct tcp_sock *tp = tcp_sk(smc->tcpsocket->sk);
+	struct smc_curs prep;
+	unsigned long max_wait_for_tx_space;
+	int copy, copied = 0, to_send = len;
+	int writespace;
+	int tx_top = 0, tx_bot = 0;
+	int rc = 0;
+	int tx_cnt_prep;
+	char *buf;
+
+	if (timer_pending(&smc->cork_timer)) {
+		del_timer_sync(&smc->cork_timer);
+		destroy_timer_on_stack(&smc->cork_timer);
+	}
+	max_wait_for_tx_space =  SMC_WAIT_TX_SPACE_TIMEO + jiffies;
+again:
+	if ((smc->sk.sk_state != SMC_ACTIVE) &&
+	    (smc->sk.sk_state != SMC_APPLCLW1))
+		return (smc->sk.sk_state == SMC_INIT
+		      ? -ENOTCONN : -ECONNRESET);
+
+	writespace = atomic_read(&conn->tx_buf_space);
+	if (!writespace) {
+		writespace = smc_wait_tx_space(smc, msg->msg_flags);
+		if (writespace < 0)
+			return ((copied && (writespace == -EAGAIN))
+			       ? copied : writespace);
+		if (!writespace) {
+			rc = smc_give_up_send(smc, copied);
+			if (rc)
+				return rc;
+			if (msg->msg_flags & MSG_DONTWAIT)
+				return copied ? copied : -EAGAIN;
+			if ((smc->sk.sk_err) ||
+			    (time_after_eq(jiffies, max_wait_for_tx_space)))
+				return copied ? copied : -EPIPE;
+			if (smc_check_peer_gone(smc->conn.lgr)) {
+				smc->sk.sk_err = EPIPE;
+				release_sock(&smc->sk);
+				smc_terminate_conn(smc->conn.lgr);
+				lock_sock(&smc->sk);
+				return copied ? copied : -EPIPE;
+			}
+			goto again;
+		}
+	}
+	if (smc->sk.sk_err ||
+	    smc->sk.sk_shutdown & SEND_SHUTDOWN)
+		return -EPIPE;
+
+	writespace = atomic_read(&conn->tx_buf_space);
+	rc = smc_give_up_send(smc, copied);
+	if (rc)
+		return rc;
+
+	if (msg->msg_flags & MSG_OOB) {
+		if (writespace < len) {
+			conn->local_tx_ctrl.p_flags.urg_data_pending = 1;
+			rc = smc_ctrl_send(conn, conn->lnk_idx);
+			conn->local_tx_ctrl.p_flags.urg_data_pending = 0;
+		}
+	}
+
+	copy = min_t(size_t, to_send, writespace);
+	/* determine start of tx_buf */
+	prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	tx_cnt_prep = prep.s.curs.c;
+	buf = conn->rmb_tx_elem->rmb_tx;
+	/* determine tx_buf pieces - bottom and top of tx_buf */
+	if (tx_cnt_prep + copy <= conn->rmb_tx_size) {
+		tx_top = 0;
+		tx_bot = copy;
+		if (memcpy_from_msg(buf + tx_cnt_prep, msg, copy))
+			return -EFAULT;
+	} else {
+		tx_bot = conn->rmb_tx_size - tx_cnt_prep;
+		tx_top = copy - tx_bot;
+		if (memcpy_from_msg(buf + tx_cnt_prep, msg, tx_bot))
+			return -EFAULT;
+		if (memcpy_from_msg(buf, msg, tx_top))
+			return -EFAULT;
+	}
+	smc_curs_add(conn->rmb_tx_size, &prep.s.curs, copy);
+	atomic64_set(&conn->tx_curs_prep.s.acurs, prep.s.lcurs);
+	smp_mb__before_atomic();
+	atomic_sub(copy, &conn->tx_buf_space);
+	smp_mb__after_atomic();
+
+	if (msg->msg_flags & MSG_OOB)
+		atomic64_set(&conn->tx_urg_curs.s.acurs, prep.s.lcurs);
+
+	if ((tp->nonagle & TCP_NAGLE_CORK || msg->msg_flags & MSG_MORE) &&
+	    (writespace > copy)) {
+		if (!timer_pending(&smc->cork_timer))
+			/* cancelled at the beginning ? */
+			mod_timer(&smc->cork_timer,
+				  jiffies + msecs_to_jiffies(200));
+	} else {
+		if (atomic_read(&conn->rx_buf_space))
+			smc_write_data(smc);
+		else if (!delayed_work_pending(&smc->write_work))
+			schedule_delayed_work(&smc->write_work, 0);
+	}
+
+	copied = copied + copy;
+	to_send = to_send - copy;
+	if (copied < len)
+		goto again;
+
+	return copied;
+}
+
+static inline void smc_fill_sge(int *num, struct tx_sge sge[], int offset1,
+				int len1, int offset2, int len2)
+{
+	memset(sge, 0, 2 * sizeof(struct tx_sge));
+	sge[0].offset = offset1;
+	sge[0].len = len1;
+	if (len2) {
+		*num = 2;
+		sge[1].offset = offset2;
+		sge[1].len = len2;
+	} else {
+		*num = 1;
+	}
+}
+
+static inline void smc_write_data_adapt(struct smc_connection *conn,
+					struct smc_curs *prod,
+					struct smc_curs *sent, int len)
+{
+	smc_curs_add(conn->peer_rx_buf_len, &prod->s.curs, len);
+	smp_mb__before_atomic();
+	atomic_sub(len, &conn->rx_buf_space);
+	smp_mb__after_atomic();
+	smc_curs_add(conn->rmb_tx_size, &sent->s.curs, len);
+}
+
+void smc_write_data(struct smc_sock *smc)
+{
+	struct smc_link *lnk;
+	struct smc_connection *conn = &smc->conn;
+	int copy, space_left1, space_left2, send_len, to_send;
+	int writespace;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs sent;
+	struct smc_curs prep;
+	struct tx_sge sge[2];
+	int tx_top = 0, tx_bot = 0;
+	int tx_top1 = 0, tx_top2 = 0;
+	int tx_bot1 = 0, tx_bot2 = 0;
+	int rc = 0, num, urgent = 0;
+	u8 lnk_idx;
+
+again:
+	lnk_idx = conn->lnk_idx;
+	if (conn->lgr->lnk_down[lnk_idx]) {
+		release_sock(&smc->sk);
+		msleep(1000);
+		lock_sock(&smc->sk);
+		goto again;
+	}
+	lnk = &conn->lgr->lnk[lnk_idx];
+	sent.s.lcurs = atomic64_read(&conn->tx_curs_sent.s.acurs);
+	prep.s.lcurs = atomic64_read(&conn->tx_urg_curs.s.acurs);
+	if (!prep.s.curs.w && !prep.s.curs.c)
+		prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	else
+		urgent = 1;
+	to_send = smc_curs_diff(conn->rmb_tx_size, &sent, &prep);
+	if (!to_send)
+		goto send_ctrl;
+	writespace = atomic_read(&conn->rx_buf_space);
+	if (writespace <= 0)
+		return;
+	if (find_first_zero_bit(lnk->pending_bit_mask, lnk->send_wr_num) ==
+		lnk->send_wr_num) {
+		schedule_delayed_work(&smc->write_work, (HZ / 10));
+		return;
+	}
+	copy = min_t(size_t, to_send, writespace);
+	if (sent.s.curs.c + copy <= conn->rmb_tx_size) {
+		tx_top = 0;
+		tx_bot = copy;
+	} else {
+		tx_bot = conn->rmb_tx_size - sent.s.curs.c;
+		tx_top = copy - tx_bot;
+	}
+	prod.s.lcurs = atomic64_read(&conn->local_tx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_rx_ctrl.c_curs.s.acurs);
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left1 = conn->peer_rx_buf_len - prod.s.curs.c;
+		space_left2 = cons.s.curs.c;
+
+		send_len = min_t(size_t, copy, space_left1);
+		if (send_len <= tx_bot) {
+			tx_bot1 = send_len;
+			tx_bot2 = tx_bot - tx_bot1;
+			tx_top1 = 0;
+			tx_top2 = tx_top;
+		} else {
+			tx_bot1 = tx_bot;
+			tx_bot2 = 0;
+			tx_top1 = send_len - tx_bot;
+			tx_top2 = tx_top - tx_top1;
+		}
+		smc_fill_sge(&num, sge, sent.s.curs.c, tx_bot1, 0, tx_top1);
+		rc = smc_rdma_write(conn, prod.s.curs.c, lnk_idx, num, sge);
+		if (rc)
+			return;
+		copy -= send_len;
+		smc_write_data_adapt(conn, &prod, &sent, send_len);
+
+		if (copy && space_left2 && (tx_bot2 + tx_top2 > 0)) {
+			send_len = min_t(size_t, copy, space_left2);
+			if (tx_bot2 > send_len) {
+				tx_bot2 = send_len;
+				tx_top2 = 0;
+			} else {
+				if (tx_bot2 + tx_top2 > send_len)
+					tx_top2 = send_len - tx_bot2;
+			}
+			if (tx_bot2)
+				smc_fill_sge(&num, sge, sent.s.curs.c,
+					     tx_bot2, tx_top1, tx_top2);
+			else if (tx_top2)
+				smc_fill_sge(&num, sge, tx_top1, tx_top2, 0, 0);
+			rc = smc_rdma_write(conn, 0, lnk_idx, num, sge);
+			if (rc)
+				return;
+			smc_write_data_adapt(conn, &prod, &sent,
+					     tx_bot2 + tx_top2);
+		}
+	} else {
+		space_left1 = cons.s.curs.c - prod.s.curs.c;
+		send_len = min_t(size_t, copy, space_left1);
+		if (send_len <= tx_bot) {
+			tx_bot = send_len;
+			tx_top = 0;
+		} else {
+			if ((send_len - tx_bot) <= tx_top)
+				tx_top = send_len - tx_bot;
+		}
+		smc_fill_sge(&num, sge, sent.s.curs.c, tx_bot, 0, tx_top);
+		rc = smc_rdma_write(conn, prod.s.curs.c, lnk_idx, num, sge);
+		if (rc)
+			return;
+		smc_write_data_adapt(conn, &prod, &sent, send_len);
+	}
+	atomic64_set(&conn->local_tx_ctrl.p_curs.s.acurs, prod.s.lcurs);
+	atomic64_set(&conn->tx_curs_sent.s.acurs, sent.s.lcurs);
+
+	/* send cursor updates and signal peer */
+	if (to_send > atomic_read(&conn->rx_buf_space))
+		conn->local_tx_ctrl.p_flags.write_blocked = 1;
+	else
+		conn->local_tx_ctrl.p_flags.write_blocked = 0;
+
+	if (urgent) {
+		conn->local_tx_ctrl.p_flags.urg_data_pending = 1;
+		conn->local_tx_ctrl.p_flags.urg_data_present = 1;
+	}
+
+send_ctrl:
+	rc = smc_ctrl_send(conn, lnk_idx);
+	if (rc)
+		/* failover is scheduled */
+		return;
+
+	if (urgent) {
+		conn->local_tx_ctrl.p_flags.urg_data_pending = 0;
+		conn->local_tx_ctrl.p_flags.urg_data_present = 0;
+		atomic64_set(&conn->tx_urg_curs.s.acurs, 0);
+			/* could be lost in case of failover */
+	}
+
+	prep.s.lcurs = atomic64_read(&conn->tx_urg_curs.s.acurs);
+	if (!prep.s.curs.w && !prep.s.curs.c)
+		prep.s.lcurs = atomic64_read(&conn->tx_curs_prep.s.acurs);
+	if (smc_curs_diff(conn->rmb_tx_size, &sent, &prep) &&
+	    (smc->sk.sk_state != SMC_CLOSED)) {
+		schedule_delayed_work(&smc->write_work, 0);
+	} else {
+		if (wq_has_sleeper(smc->sk.sk_wq) &&
+		    atomic_read(&conn->tx_buf_space) &&
+		    test_bit(SOCK_NOSPACE, &smc->sk.sk_socket->flags))
+			smc_sock_wake_tx(&smc->sk);
+		if (smc->sk.sk_state != SMC_ACTIVE)
+			wake_up(&smc->prep_waiter);
+	}
+}
+
+void smc_write_worker(struct work_struct *work)
+{
+	int rc;
+
+	struct smc_sock *smc = container_of(work, struct smc_sock,
+		write_work.work);
+
+	lock_sock(&smc->sk);
+again:
+	if (smc_prepared_sends(&smc->conn) == 0)
+		goto out;
+	rc = smc_wait_rx_space(smc);
+	if (rc < 0)
+		goto out;
+	if (!rc) {
+		if (!smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close &&
+		    !smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close &&
+		    (smc->sk.sk_state != SMC_CLOSED))
+			goto again;
+		else
+			goto out;
+	}
+	if (!smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close &&
+	    !smc->conn.local_rx_ctrl.conn_state_flags.abnormal_close &&
+	    (smc->sk.sk_state != SMC_CLOSED))
+		smc_write_data(smc);
+out:
+	release_sock(&smc->sk);
+}
+
+void smc_send_close_alt(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int i;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((atomic_read(&lgr->lnk[i].state) == SMC_LINK_UP) &&
+		    (i != smc->conn.lnk_idx))
+			break;
+	}
+	if (i <= SMC_MAX_SYM_LINKS)
+		smc_ctrl_send(&smc->conn, i);
+}
+
+int smc_send_close(struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = smc->conn.lgr;
+	int rc;
+
+	if (smc_to_read(smc) > 0)
+		smc->conn.local_tx_ctrl.conn_state_flags.abnormal_close = 1;
+	else
+		smc->conn.local_tx_ctrl.conn_state_flags.closed_conn = 1;
+	if (lgr->lgr_type != NONE) {
+		rc = smc_ctrl_send(&smc->conn, smc->conn.lnk_idx);
+		if (rc && (lgr->lgr_type != SINGLE)) {
+			/* try other link */
+			smc_send_close_alt(smc);
+		}
+	}
+	wake_up_interruptible(&smc->rx_waiter);
+	return 0;
+}
+
+int smc_to_read(struct smc_sock *smc)
+{
+	int rc;
+	struct smc_curs prod; /* to receive */
+	struct smc_curs cons; /* already consumed */
+	struct smc_curs spl_cons; /* spliced data in flight */
+	struct smc_curs urg;
+
+	if ((smc->conn.rx_urg_state != SMC_URG_VALID) &&
+	    !smc_splice_in_use(smc))
+		return atomic_read(&smc->conn.bytes_to_rcv);
+
+	/* local rx cursor consumer_c is up to date */
+	prod.s.lcurs = atomic64_read(&smc->conn.local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+
+	if ((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&smc->conn.rx_urg_curs.s.acurs);
+		if (smc_curs_diff(smc->conn.rmb_rx_size, &cons, &urg))
+			prod.s.lcurs =
+				atomic64_read(&smc->conn.rx_urg_curs.s.acurs);
+	}
+	if (smc_splice_in_use(smc)) {
+		spl_cons.s.lcurs =
+			atomic64_read(&smc->conn.local_rx_spliced.s.acurs);
+		if (spl_cons.s.curs.reserved0)
+			cons = spl_cons;
+	}
+
+	rc = smc_curs_diff(smc->conn.rmb_rx_size, &cons, &prod);
+	return (rc == -EALREADY) ? 0 : rc;
+}
+
+static void smc_update_cons_curs(struct smc_sock *smc, struct smc_link *lnk)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0, to_confirm;
+	struct smc_curs cfed;
+	struct smc_curs cons;
+
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	cfed.s.lcurs = atomic64_read(&conn->rx_curs_confirmed.s.acurs);
+	to_confirm = smc_curs_diff(conn->rmb_rx_size, &cfed, &cons);
+
+	if (conn->local_rx_ctrl.p_flags.cons_curs_upd_req ||
+	    ((to_confirm > conn->upd_cons_limit) &&
+	     ((to_confirm / (conn->rmb_rx_size / 2) > 0) ||
+	      conn->local_rx_ctrl.p_flags.write_blocked ||
+	      conn->local_rx_ctrl.p_flags.urg_data_pending))) {
+		if (lnk)
+			rc = smc_ctrl_send(conn, lnk - &conn->lgr->lnk[0]);
+		else
+			rc = smc_ctrl_send(conn, conn->lnk_idx);
+		if (rc)
+			return;
+		atomic64_set(&conn->rx_curs_confirmed.s.acurs,
+			    atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs));
+		conn->local_rx_ctrl.p_flags.cons_curs_upd_req = 0;
+	}
+	if (conn->local_rx_ctrl.p_flags.write_blocked &&
+	    (!smc_to_read(smc) > 0))
+		conn->local_rx_ctrl.p_flags.write_blocked = 0;
+}
+
+static void smc_pipe_buf_noop(struct pipe_inode_info *pipe,
+			      struct pipe_buffer *buf)
+{
+	;
+}
+
+static int smc_pipe_buf_steal(struct pipe_inode_info *pipe,
+			      struct pipe_buffer *buf)
+{
+	return 1;
+}
+
+struct smc_pipe_buf_priv {
+	struct smc_sock *smc;
+	unsigned int     len;
+};
+
+static void smc_pipe_buf_release(struct pipe_inode_info *pipe,
+				 struct pipe_buffer *buf)
+{
+	struct smc_curs curs;
+	struct smc_pipe_buf_priv *priv
+		= (struct smc_pipe_buf_priv *)buf->private;
+	struct smc_sock *smc = priv->smc;
+
+	/* Update cursors when we reached the last page */
+	curs.s.lcurs = atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+	smc_curs_add(smc->conn.rmb_rx_size, &curs.s.curs, priv->len);
+	atomic64_set(&smc->conn.local_tx_ctrl.c_curs.s.acurs, curs.s.lcurs);
+	smc_update_cons_curs(smc, NULL);
+	wake_up(&smc->splice_waiter);
+	kfree((void *)buf->private);
+}
+
+static const struct pipe_buf_operations smc_pipe_ops = {
+	.can_merge = 0,
+	.confirm = generic_pipe_buf_confirm,
+	.get = smc_pipe_buf_noop,
+	.release = smc_pipe_buf_release,
+	.steal = smc_pipe_buf_steal
+};
+
+static void smc_splice_spd_release(struct splice_pipe_desc *spd, unsigned int i)
+{
+	;
+}
+
+static void smc_free_spd(struct smc_sock *smc)
+{
+	if (smc->spd) {
+		kfree(smc->spd->partial);
+		smc->spd->partial = NULL;
+		kfree(smc->spd->pages);
+		smc->spd->pages = NULL;
+		kfree(smc->spd);
+		smc->spd = NULL;
+	}
+}
+
+static int smc_alloc_spd(struct smc_sock *smc)
+{
+	int msize = smc->conn.rmb_rx_size / PAGE_SIZE + 1;
+
+	smc->spd = kzalloc(sizeof(*smc->spd), GFP_KERNEL);
+	if (!smc->spd)
+		return -ENOMEM;
+	smc->spd->pages = kcalloc(msize, sizeof(struct page), GFP_KERNEL);
+	if (!smc->spd->pages)
+		goto error;
+	smc->spd->partial = kcalloc(msize, sizeof(struct partial_page),
+				    GFP_KERNEL);
+	if (!smc->spd->partial)
+		goto error;
+	smc->spd->nr_pages_max = msize;
+	smc->spd->ops = &smc_pipe_ops;
+	smc->spd->spd_release = smc_splice_spd_release;
+
+	return 0;
+
+error:
+	smc_free_spd(smc);
+	return -ENOMEM;
+}
+
+/* no extra handling of splice options required:
+ *   SPLICE_F_MOVE doesn't apply and can be ignored
+ *   SPLICE_F_NONBLOCK is handled by splice_to_pipe
+ *   SPLICE_F_MORE doesn't apply (fdout is a pipe)
+ *   SPLICE_F_GIFT is unused
+ */
+static int smc_splice_to_pipe(struct smc_sock *smc,
+			      struct pipe_inode_info *pipe,
+			      unsigned int flags, void *src, int len)
+{
+	struct partial_page *ppage;
+	struct splice_pipe_desc *spd;
+	struct smc_pipe_buf_priv *priv;
+	int rc, i;
+
+	if (!len)
+		return 0;
+	if (!smc->spd && smc_alloc_spd(smc))
+		return -ENOMEM;
+	spd = smc->spd;
+	spd->nr_pages = 0;
+	spd->flags = flags;
+
+	while (len) {
+		spd->pages[spd->nr_pages] = virt_to_page(src);
+		ppage = &spd->partial[spd->nr_pages];
+		ppage->offset = (unsigned long)src & ~PAGE_MASK;
+		ppage->len = min_t(int, PAGE_SIZE - ppage->offset, len);
+		src += ppage->len;
+		len -= ppage->len;
+		priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+		if (!priv)
+			goto error;
+		priv->smc = smc;
+		priv->len = ppage->len;
+		ppage->private = (unsigned long)priv;
+		spd->nr_pages++;
+	}
+
+	rc = splice_to_pipe(pipe, spd);
+	if (rc < 0 && smc->sk.sk_err)
+		goto error;
+	return rc;
+
+error:
+	for (i = 0; i < spd->nr_pages; ++i)
+		kfree((void *)spd->partial[i].private);
+	return -EFAULT;
+}
+
+/* Wait for data to arrive
+ *   @smc    smc socket
+ *   @len    num bytes to wait for
+ *   @wall   don't return until full request is fulfilled
+ *   @timeo  max seconds to wait, 0 for no timeout
+ * Returns boolean value or -EAGAIN in case timeout expired.
+ */
+int smc_wait_rx_data(struct smc_sock *smc, int len, bool wall,
+		     long timeo)
+{
+	int rc;
+	struct sock *sk = &smc->sk;
+	struct smc_connection *conn = &smc->conn;
+	int target;
+	DEFINE_WAIT(wait);
+
+	if ((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID)
+		target = 1;
+	else
+		target = sock_rcvlowat(sk, wall, len);
+	if (smc_to_read(smc) >= target)
+		return 1;
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+	set_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
+	rc = sk_wait_event(sk, &timeo,
+			   (sk->sk_err != 0) ||
+			   sk->sk_shutdown & RCV_SHUTDOWN ||
+			   sock_flag(sk, SOCK_DONE) ||
+			   (smc_to_read(smc) >= target) ||
+			   smc_stop_received(conn));
+	clear_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
+	finish_wait(sk_sleep(sk), &wait);
+	return ((rc || timeo) ? rc : -EAGAIN);
+}
+
+static inline void smc_skip_urg_data(struct smc_sock *smc,
+				     struct smc_curs *cons,
+				     struct smc_curs *urg)
+{
+	if (((smc->conn.rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) &&
+	    (cons->s.curs.c == urg->s.curs.c) &&
+	    (cons->s.curs.w == urg->s.curs.w))
+		smc->conn.rx_urg_state |= SMC_URG_RECV;
+}
+
+/* derived from tcp_recv_urg */
+static int smc_conn_recv_urg(struct smc_sock *smc, struct msghdr *msg,
+			     size_t len, unsigned int flags)
+{
+	int rc = 0;
+
+	if (sock_flag(&smc->sk, SOCK_URGINLINE) ||
+	    !(smc->conn.rx_urg_state & SMC_URG_VALID) ||
+	    (smc->conn.rx_urg_state & SMC_URG_READ))
+		return -EINVAL;
+
+	if (!(flags & MSG_PEEK)) {
+		smc->conn.rx_urg_state |= SMC_URG_READ;
+		atomic_dec(&smc->conn.bytes_to_rcv);
+	}
+
+	msg->msg_flags |= MSG_OOB;
+
+	if (len > 0) {
+		if (!(flags & MSG_TRUNC))
+			rc = memcpy_to_msg(msg, &smc->conn.rx_urg_data, 1);
+		len = 1;
+	} else {
+		msg->msg_flags |= MSG_TRUNC;
+	}
+	if (!sock_flag(&smc->sk, SOCK_URGINLINE)) {
+		struct smc_curs cons;
+
+		/* may cause smc_curs_diff errors, which are acceptable here */
+		cons.s.lcurs =
+			atomic64_read(&smc->conn.local_tx_ctrl.c_curs.s.acurs);
+		smc_curs_add(smc->conn.rmb_rx_size, &cons.s.curs, 1);
+		atomic64_set(&smc->conn.local_tx_ctrl.c_curs.s.acurs,
+			     cons.s.lcurs);
+	}
+
+	return rc ? -EFAULT : len;
+}
+
+int smc_conn_recv(struct smc_sock *smc, struct msghdr *msg, size_t len,
+		  int flags)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0;
+	int toread = 0, space_left, copy, buffer_len;
+	int target;		/* Read at least these many bytes */
+	char *buffer;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs urg;
+	struct smc_curs spl_cons;
+	long timeo;
+	int loop = 0;
+
+	msg->msg_namelen = 0;
+
+	if (smc->sk.sk_state == SMC_INIT)
+		return -ENOTCONN;
+
+	buffer_len = conn->rmb_rx_size;
+	buffer =  conn->rmb_rx_elem->rmb_rx;
+	len = min_t(size_t, len, conn->rmb_rx_size);
+
+	if (flags & MSG_OOB)
+		return smc_conn_recv_urg(smc, msg, len, flags);
+
+again:
+	target = sock_rcvlowat(&smc->sk, flags & MSG_WAITALL, len);
+	timeo = sock_rcvtimeo(&smc->sk, flags & MSG_DONTWAIT);
+	if (signal_pending(current)) {
+		rc = timeo ? -EINTR : -EAGAIN;
+		return rc;
+	}
+	rc = smc_wait_rx_data(smc, target, flags & MSG_WAITALL, timeo);
+	if ((rc == -EAGAIN) || (rc == -EINTR))
+		return rc;
+	if (!rc) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			release_sock(&smc->sk);
+			smc_terminate_conn(smc->conn.lgr);
+			lock_sock(&smc->sk);
+			return -EINTR;
+		}
+		goto again;
+	}
+	if (smc_splice_in_use(smc)) {
+		if (smc_splice_in_progress(smc) && (flags &  MSG_DONTWAIT))
+			return -EAGAIN;
+		wait_event_interruptible(smc->splice_waiter,
+					 !smc_splice_in_progress(smc));
+	}
+	toread = smc_to_read(smc);
+	rc = 0;
+
+	if ((toread <= 0) &&
+	    (smc->sk.sk_err ||
+	     smc->sk.sk_shutdown & RCV_SHUTDOWN ||
+	     sock_flag(&smc->sk, SOCK_DONE) ||
+	     smc_stop_received(conn)))
+		return sock_error(&smc->sk);
+
+	prod.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	if (toread <= 0)
+		goto check_repeat;
+
+	if ((conn->rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+		if (smc_curs_diff(conn->rmb_rx_size, &cons, &urg))
+			prod = urg;
+		else /* data preceding urg already received */
+			conn->rx_urg_state |= SMC_URG_RECV;
+	} else {
+		urg.s.lcurs = 0;
+	}
+
+	if ((toread < target) && !smc_stop_received(conn))
+		goto check_repeat;
+
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left = prod.s.curs.c - cons.s.curs.c;
+		copy = min_t(size_t, len, space_left);
+		if (!(flags & MSG_TRUNC)) {
+			if (memcpy_to_msg(msg, buffer + cons.s.curs.c, copy))
+				return -EFAULT;
+		}
+		len -= copy;
+		rc += copy;
+	} else {
+		space_left = buffer_len - cons.s.curs.c;
+		if (space_left) {
+			copy = min_t(size_t, len, space_left);
+			if (!(flags & MSG_TRUNC)) {
+				if (memcpy_to_msg(msg, buffer + cons.s.curs.c,
+						  copy))
+					return -EFAULT;
+			}
+			len -= copy;
+			rc += copy;
+		}
+		if (len) {
+			space_left = prod.s.curs.c;
+			copy = min_t(size_t, len, space_left);
+			if (!(flags & MSG_TRUNC)) {
+				if (memcpy_to_msg(msg, buffer, copy))
+					return -EFAULT;
+			}
+			len -= copy;
+			rc += copy;
+		}
+	}
+
+	/* update cursors */
+	if (!(flags &  MSG_PEEK)) {
+		smc_curs_add(buffer_len, &cons.s.curs, rc);
+		smc_skip_urg_data(smc, &cons, &urg);
+		smp_mb__before_atomic();
+		atomic_sub(rc, &conn->bytes_to_rcv);
+		smp_mb__after_atomic();
+		atomic64_set(&conn->local_tx_ctrl.c_curs.s.acurs, cons.s.lcurs);
+		spl_cons.s.lcurs =
+			atomic64_read(&conn->local_rx_spliced.s.acurs);
+		if (spl_cons.s.curs.reserved0)
+			atomic64_set(&conn->local_rx_spliced.s.acurs,
+				     cons.s.lcurs);
+		/* send consumer cursor update if required */
+		smc_update_cons_curs(smc, NULL);
+	}
+check_repeat:
+	if ((toread < target) &&
+	    !smc_stop_received(conn) &&
+	    !conn->local_tx_ctrl.conn_state_flags.abnormal_close) {
+		loop = 0;
+		goto again;
+	}
+
+	return rc;
+}
+
+ssize_t smc_conn_splice_read(struct smc_sock *smc, struct pipe_inode_info *pipe,
+			     size_t len, unsigned int flags)
+{
+	struct smc_connection *conn = &smc->conn;
+	int rc = 0, ret = 0;
+	int toread = 0, space_left, copy, buffer_len;
+	int target = 1;		/* read at least these many bytes */
+	char *buffer;
+	struct smc_curs prod;
+	struct smc_curs cons;
+	struct smc_curs spl_cons;
+	struct smc_curs urg;
+	long timeo;
+	int loop = 0;
+
+	if (smc->sk.sk_state == SMC_INIT)
+		return -ENOTCONN;
+
+	buffer_len = conn->rmb_rx_size;
+	buffer =  conn->rmb_rx_elem->rmb_rx;
+	len = min_t(size_t, len, conn->rmb_rx_size);
+
+again:
+	timeo = sock_rcvtimeo(&smc->sk, 0);
+	if (signal_pending(current)) {
+		rc = timeo ? -EINTR : -EAGAIN;
+		return rc;
+	}
+	rc = smc_wait_rx_data(smc, target, 0, timeo);
+	if ((rc == -EAGAIN) || (rc == -EINTR))
+		return rc;
+	if (!rc) {
+		if (smc_check_peer_gone(smc->conn.lgr)) {
+			release_sock(&smc->sk);
+			smc_terminate_conn(smc->conn.lgr);
+			lock_sock(&smc->sk);
+			return -EINTR;
+		}
+		goto again;
+	}
+	toread = smc_to_read(smc);
+
+	if ((toread <= 0) &&
+	    (smc->sk.sk_err ||
+	     smc->sk.sk_shutdown & RCV_SHUTDOWN ||
+	     sock_flag(&smc->sk, SOCK_DONE) ||
+	     smc_stop_received(conn)))
+		return sock_error(&smc->sk);
+
+	rc = 0;
+	prod.s.lcurs = atomic64_read(&conn->local_rx_ctrl.p_curs.s.acurs);
+	cons.s.lcurs = atomic64_read(&conn->local_tx_ctrl.c_curs.s.acurs);
+	if (toread <= 0)
+		goto check_repeat;
+
+	if ((conn->rx_urg_state & SMC_URG_MASK) == SMC_URG_VALID) {
+		urg.s.lcurs = atomic64_read(&conn->rx_urg_curs.s.acurs);
+		if (smc_curs_diff(conn->rmb_rx_size, &cons, &urg))
+			prod = urg;
+		else /* data preceding urg already received */
+			conn->rx_urg_state |= SMC_URG_RECV;
+	} else {
+		urg.s.lcurs = 0;
+	}
+
+	if ((toread < target) && !smc_stop_received(conn))
+		goto check_repeat;
+
+	if (prod.s.curs.w == cons.s.curs.w) {
+		space_left = prod.s.curs.c - cons.s.curs.c;
+		copy = min_t(size_t, len, space_left);
+		rc = smc_splice_to_pipe(smc, pipe, flags,
+					buffer + cons.s.curs.c, copy);
+		if (rc < 0)
+			return -EFAULT;
+		len -= rc;
+	} else {
+		space_left = buffer_len - cons.s.curs.c;
+		if (space_left) {
+			copy = min_t(size_t, len, space_left);
+			rc = smc_splice_to_pipe(smc, pipe, flags,
+						buffer + cons.s.curs.c,
+						copy);
+			if (rc < 0)
+				return -EFAULT;
+			len -= rc;
+		}
+		if (len) {
+			space_left = prod.s.curs.c;
+			copy = min_t(size_t, len, space_left);
+			ret = smc_splice_to_pipe(smc, pipe, flags, buffer,
+						 copy);
+			if (ret < 0)
+				return -EFAULT;
+			len -= ret;
+			rc += ret;
+		}
+	}
+
+	/* update cursors */
+	spl_cons.s.lcurs = atomic64_read(&conn->local_rx_spliced.s.acurs);
+	if (!spl_cons.s.curs.reserved0) {
+		/* no previous splice in progress, initialize to current */
+		spl_cons = cons;
+		spl_cons.s.curs.reserved0 = 1;
+	}
+	smc_curs_add(buffer_len, &spl_cons.s.curs, rc);
+	smc_skip_urg_data(smc, &spl_cons, &urg);
+	atomic64_set(&conn->local_rx_spliced.s.acurs, spl_cons.s.lcurs);
+	/* we update cursors in a callback after splicing completed */
+check_repeat:
+	if ((toread < target) &&
+	    !smc_stop_received(conn) &&
+	    !conn->local_tx_ctrl.conn_state_flags.abnormal_close) {
+		loop = 0;
+		goto again;
+	}
+
+	return rc;
+}
+
+void smc_check_qp_attr(struct smc_link *lnk)
+{
+	int rc;
+	struct ib_qp_attr *attr = &lnk->qp_attr;
+	struct ib_qp_init_attr init_attr;
+
+	memset(attr, 0, sizeof(struct ib_qp_attr));
+	memset(&init_attr, 0, sizeof(struct ib_qp_init_attr));
+	rc = ib_query_qp(lnk->roce_qp, attr,
+			 IB_QP_STATE |
+			 IB_QP_CUR_STATE |
+			 IB_QP_PKEY_INDEX |
+			 IB_QP_PORT |
+			 IB_QP_QKEY |
+			 IB_QP_AV |
+			 IB_QP_PATH_MTU |
+			 IB_QP_TIMEOUT |
+			 IB_QP_RETRY_CNT |
+			 IB_QP_RNR_RETRY |
+			 IB_QP_RQ_PSN |
+			 IB_QP_ALT_PATH |
+			 IB_QP_MIN_RNR_TIMER |
+			 IB_QP_SQ_PSN |
+			 IB_QP_PATH_MIG_STATE |
+			 IB_QP_CAP |
+			 IB_QP_VID |
+			 IB_QP_DEST_QPN,
+			 &init_attr);
+
+	lnk->send_wr_num = min_t(size_t, smc_ctrl_buf_cnt,
+				 lnk->qp_attr.cap.max_send_wr);
+	lnk->recv_wr_num = min_t(size_t, smc_ctrl_buf_cnt * 10,
+				 lnk->qp_attr.cap.max_recv_wr);
+}
+
+void smc_check_port_attr(struct smc_ib_device *smc_ibdev, int port_num)
+{
+	struct ib_device *dev = smc_ibdev->dev;
+	struct ib_port_attr *pattr;
+	int rc;
+
+	pattr = &smc_ibdev->pattr[port_num];
+	memset(pattr, 0, sizeof(struct ib_port_attr));
+	rc = ib_query_port(dev, port_num + 1, pattr);
+}
+
+void smc_check_dev_attr(struct smc_ib_device *smc_ibdev)
+{
+	struct ib_device *dev = smc_ibdev->dev;
+	struct ib_device_attr *attr = &smc_ibdev->attr;
+	int i, rc;
+
+	memset(attr, 0, sizeof(struct ib_device_attr));
+	rc = ib_query_device(dev, attr);
+
+	for (i = 0; i < 2; i++)
+		smc_check_port_attr(smc_ibdev, i);
+}
+
diff --git a/net/smc/smc_llc.c b/net/smc/smc_llc.c
new file mode 100644
index 0000000..7f6eb2a
--- /dev/null
+++ b/net/smc/smc_llc.c
@@ -0,0 +1,1597 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  SMC-R Link Layer Control
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Klaus Wacker <Klaus.Wacker@de.ibm.com>
+ *              Ursula Braun <ursula.braun@de.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/wait.h>
+#include <linux/kthread.h>
+#include <linux/hardirq.h>
+#include <linux/types.h>
+#include <net/tcp.h>
+#include <net/sock.h>
+#ifdef CONFIG_S390
+#include <asm/debug.h>
+#endif
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <linux/splice.h>
+#include <linux/mm.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+static enum llc_msg_group llc_msg_to_grp(enum llc_msg_type type)
+{
+	enum llc_msg_group grp;
+
+	switch (type) {
+	case LLC_ADD_LINK:
+		grp = LLC_GRP_ADD_LINK;
+		break;
+	case LLC_DEL_LINK:
+		grp = LLC_GRP_DEL_LINK;
+		break;
+	case LLC_CONFIRM_RKEY:
+		grp = LLC_GRP_CONF_RKEY;
+		break;
+	case LLC_TEST_LINK:
+		grp = LLC_GRP_TEST_LINK;
+		break;
+	default:
+		grp = LLC_GRP_NONE;
+	}
+	return grp;
+}
+
+int llc_enqueue(struct smc_link *lnk, struct llc_msg *msg)
+{
+	struct smc_link_group *lgr;
+	struct llc_qentry *qentry;
+	unsigned long flags;
+
+	lgr = lnk->lnk_grp;
+	qentry = kzalloc(sizeof(*qentry), GFP_ATOMIC);
+	if (!qentry)
+		return -ENOMEM;
+	qentry->link = lnk;
+	memcpy(&qentry->msg, msg, sizeof(struct llc_msg));
+	if (msg->hd.flags & LLC_FLAG_RESP) {
+		switch (msg->hd.type) {
+		case LLC_ADD_LINK:
+		case LLC_DEL_LINK:
+		case LLC_CONFIRM_LINK:
+		case LLC_ADD_LINK_CONT:
+			lgr->llc_ctl[LLC_SERV].qentry = qentry;
+			break;
+		case LLC_CONFIRM_RKEY:
+		case LLC_CONFIRM_RKEY_CONT:
+		case LLC_DELETE_RKEY:
+			lgr->llc_ctl[lgr->role].qentry = qentry;
+			break;
+		case LLC_TEST_LINK:
+			lgr->llc_ctl[LLC_TESTLINK].qentry = qentry;
+			break;
+		default:
+			llc_send_del_link(lnk, NULL, 0, LLC_DEL_PROT_VIOL,
+					  LLC_NO_NOTIFY);
+			smc_terminate_conn(lgr);
+			kfree(qentry);
+			break;
+		}
+		wake_up_interruptible(&lgr->llc_waiter);
+		return 0;
+	}
+	spin_lock_irqsave(&lgr->llc_event_q_lock, flags);
+	list_add_tail(&qentry->list, &lgr->llc_event_q);
+	spin_unlock_irqrestore(&lgr->llc_event_q_lock, flags);
+	if (lgr->llc_wq)
+		queue_work(lgr->llc_wq, &lgr->llc_event_work);
+
+	return 0;
+}
+
+int llc_initiate(struct smc_link_group *lgr, enum llc_msg_group llc_grp)
+{
+	enum llc_msg_group llc_grp_allowed;
+	int peer;
+	enum llc_ctl_ind llc_seq;
+	int rest_time = 1;
+
+	llc_grp_allowed = (llc_grp == LLC_GRP_CONF_RKEY) ?
+				LLC_GRP_CONF_RKEY : LLC_GRP_NONE;
+	peer = !lgr->role;
+	switch (llc_grp) {
+	case LLC_GRP_CONF_RKEY:
+		llc_seq = (enum llc_ctl_ind)lgr->role;
+		break;
+	case LLC_GRP_TEST_LINK:
+		llc_seq = 2;
+		peer = 2;
+		llc_grp_allowed = LLC_GRP_TEST_LINK;
+		break;
+	default:
+		llc_seq = LLC_SERV;
+	}
+reserve_llc:
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	if ((lgr->llc_ctl[llc_seq].active == LLC_GRP_NONE) &&
+	    (lgr->llc_ctl[peer].active <= llc_grp_allowed)) {
+		lgr->llc_ctl[llc_seq].active = llc_grp;
+		spin_unlock_bh(&lgr->llc_ctl_lock);
+		return rest_time;
+	}
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	rest_time = wait_event_interruptible_timeout(lgr->llc_waiter,
+		((lgr->llc_ctl[llc_seq].active == LLC_GRP_NONE) &&
+		 (lgr->llc_ctl[peer].active <= llc_grp_allowed)),
+		LLC_WAIT_TIMEO);
+	if (!rest_time)
+		return rest_time;
+	goto reserve_llc;
+}
+
+void llc_stop(struct smc_link_group *lgr, int llc_grp)
+{
+	enum llc_ctl_ind llc_seq;
+
+	switch (llc_grp) {
+	case LLC_GRP_CONF_RKEY:
+		llc_seq = (enum llc_ctl_ind)lgr->role;
+		break;
+	case LLC_GRP_TEST_LINK:
+		llc_seq = 2;
+		break;
+	default:
+		llc_seq = LLC_SERV;
+	}
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	memset(&lgr->llc_ctl[llc_seq], 0, sizeof(struct llc_ctl));
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	if (lgr->delayed_q &&
+	    (llc_grp != LLC_GRP_TEST_LINK)) {
+		queue_work(lgr->llc_wq, &lgr->llc_event_work);
+	} else {
+		wake_up_interruptible(&lgr->llc_waiter);
+	}
+}
+
+struct llc_qentry *llc_wait(struct smc_link_group *lgr, enum smc_role initiator,
+			    int time_out, u8 exp_msg)
+{
+	struct llc_ctl *ctl = &lgr->llc_ctl[initiator];
+
+	wait_event_interruptible_timeout(lgr->llc_waiter,
+					 (ctl->qentry ||
+					 ((ctl->ctrl_elem == LLC_SEND_POSTED) &&
+					  ctl->wc_status) ||
+					 (lgr->lgr_type == NONE)),
+					 time_out);
+	if (IS_ERR(ctl->qentry) ||	/* waiter has been killed */
+	    !ctl->qentry) {
+		ctl->qentry = NULL;
+		goto out;
+	}
+	if (((initiator == SMC_SERV) ||
+	     (exp_msg != LLC_CONFIRM_LINK)) &&
+	    (ctl->ctrl_elem != LLC_SEND_POSTED)) {
+		/* cancel post req when send completes after llc msg arrival */
+		lgr->lnk[ctl->lnk_idx].pending[ctl->ctrl_elem].post_ctl =
+			LLC_NO_NOTIFY; /* mark as cancelled */
+	}
+	if ((exp_msg) && (ctl->qentry->msg.hd.type != exp_msg)) {
+		if ((exp_msg == LLC_ADD_LINK) &&
+		    (ctl->qentry->msg.hd.type == LLC_DEL_LINK)) {
+			schedule_work(&lgr->llc_del_link_work);
+			return NULL;
+		}
+		kfree(ctl->qentry);
+		ctl->qentry = NULL;
+	}
+out:
+	return ctl->qentry;
+}
+
+static int llc_add_pending_send(struct smc_link *link,
+				enum llc_ctl_ind ctl_idx, u32 *elem)
+{
+	u64 wr_id;
+	int rc = 0;
+
+	wr_id = (u64)atomic64_inc_return((atomic64_t *)&link->wr_id_send);
+	rc = smc_get_ctrl_buf(link, elem);
+	if (rc < 0)
+		return rc;
+	memset(&link->pending[*elem], 0, sizeof(struct pending_send));
+	link->pending[*elem].wr_id_send = wr_id;
+	link->pending[*elem].post_ctl = ctl_idx;
+	if (ctl_idx != LLC_NO_NOTIFY)
+		link->lnk_grp->llc_ctl[ctl_idx].ctrl_elem = *elem;
+	link->send_wr[*elem].wr_id = wr_id;
+	return 0;
+}
+
+static inline int llc_send_msg(struct smc_link *link, struct llc_msg *msg,
+			       enum llc_ctl_ind ctl_idx)
+{
+	struct smc_link_group *lgr;
+	u32 index = link->send_wr_num;
+	int rc;
+
+	if ((atomic_read(&link->state) != SMC_LINK_UP) || !link->lnk_grp)
+		return -EPIPE;
+	lgr = link->lnk_grp;
+	if (lgr->lgr_type == NONE)
+		return -EPIPE;
+	if (ctl_idx != LLC_NO_NOTIFY)
+		lgr->llc_ctl[ctl_idx].lnk_idx = link - &lgr->lnk[0];
+	/* send llc message */
+	rc = llc_add_pending_send(link, ctl_idx, &index);
+	if (!rc)
+		rc = smc_wr_send(link, NULL, (u64)msg, index);
+	if (rc) {
+		if (index < link->send_wr_num)
+			test_and_clear_bit(index, link->pending_bit_mask);
+	}
+	return rc;
+}
+
+int llc_send_confirm_link(struct smc_link *link, u8 mac[], union ib_gid *gid,
+			  enum llc_reqresp reqresp, enum llc_ctl_ind ctl_idx)
+{
+	struct llc_confirm_msg confllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	u32 index;
+	int rc = 0;
+
+	memset(&confllc, 0, sizeof(confllc));
+	confllc.hd.type = LLC_CONFIRM_LINK;
+	confllc.hd.length = sizeof(struct llc_confirm_msg);
+	if (reqresp == LLC_RESP)
+		confllc.hd.flags = 0x80;
+	else
+		confllc.hd.flags = 0x00;
+	confllc.hd.flags |= SMC_CACHE_ALIGN_SIZE_SHORT;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	memcpy(confllc.sender_mac, mac, 6);
+	memcpy(confllc.sender_gid, gid, 16);
+	confllc.max_links = lgr->max_links;
+	confllc.link_num = link->link_id;
+	confllc.link_uid = htonl(lgr->lgr_id + link->link_id);
+
+	hton_three(link->roce_qp->qp_num, confllc.sender_qp_num);
+	/* send llc message */
+	if (ctl_idx != LLC_NO_NOTIFY)
+		lgr->llc_ctl[ctl_idx].lnk_idx = link - &lgr->lnk[0];
+	rc = llc_add_pending_send(link, ctl_idx, &index);
+	if (!rc)
+		rc = smc_wr_send(link, NULL, (u64)&confllc, index);
+	return rc;
+}
+
+int llc_send_add_link(struct smc_link *link, struct smc_roce_defs *rocdefs,
+		      struct smc_link *linkw, enum llc_reqresp reqresp,
+		      enum llc_ctl_ind ctl_idx)
+{
+	struct llc_add_link_msg addllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int rc = 0;
+
+	memset(&addllc, 0, sizeof(addllc));
+	addllc.hd.type = LLC_ADD_LINK;
+	addllc.hd.length = sizeof(struct llc_add_link_msg);
+	if (reqresp == LLC_RESP) {
+		addllc.hd.flags = 0x80;
+		addllc.flags2 = min(linkw->path_mtu, linkw->mtu_peer);
+	} else {
+		addllc.hd.flags = 0x00;
+		if (linkw)
+			addllc.flags2 = linkw->path_mtu;
+		else
+			addllc.flags2 = link->path_mtu;
+		if (lgr->role == SMC_SERV)
+			lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	}
+	memcpy(addllc.sender_mac, &rocdefs->mac, 6);
+	memcpy(addllc.sender_gid, &rocdefs->gid, 16);
+	if (linkw) {
+		addllc.link_num = linkw->link_id;
+		hton_three(linkw->roce_qp->qp_num, addllc.sender_qp_num);
+		hton_three(linkw->psn_initial, addllc.initial_psn);
+	}
+	/* send llc message */
+	rc = llc_send_msg(link, (struct llc_msg *)&addllc, ctl_idx);
+	return rc;
+}
+
+int llc_send_del_link(struct smc_link *link, struct smc_link *link2,
+		      u8 ord_flag, u32 reason, enum llc_ctl_ind ctl_idx)
+{
+	struct llc_del_link_msg delllc;
+
+	memset(&delllc, 0, sizeof(delllc));
+	delllc.hd.type = LLC_DEL_LINK;
+	delllc.hd.length = sizeof(struct llc_del_link_msg);
+	delllc.hd.flags = ord_flag;
+	if (link2) {
+		delllc.link_num = link2->link_id;
+	} else {
+		delllc.link_num = 0;
+		delllc.hd.flags |= LLC_FLAG_DEL_ALL;
+	}
+	delllc.reason = htonl(reason);
+
+	return llc_send_msg(link, (struct llc_msg *)&delllc, ctl_idx);
+}
+
+int llc_do_confirm_rkey(struct smc_link *link, struct rmb_rx_addrs *rmb_rx_elem)
+{
+	struct llc_confirm_rkey_msg confrkeyllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int prim_link_id, rt_idx, i, prim_lnk_idx;
+
+	memset(&confrkeyllc, 0, sizeof(confrkeyllc));
+	confrkeyllc.hd.type = LLC_CONFIRM_RKEY;
+	confrkeyllc.hd.length = sizeof(struct llc_confirm_rkey_msg);
+	confrkeyllc.hd.flags = 0x00;
+	lgr->llc_ctl[lgr->role].qentry = NULL;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	confrkeyllc.rt[0].rmb_key =
+		htonl(rmb_rx_elem->mr_rx[prim_lnk_idx]->rkey);
+	confrkeyllc.rt[0].rmb_vaddr =
+		cpu_to_be64((u64)rmb_rx_elem->rmb_rx_dma[prim_lnk_idx]);
+
+	prim_link_id = link->link_id;
+	rt_idx = 1;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if ((lgr->lnk[i].link_id == prim_link_id) ||
+		    (atomic_read(&lgr->lnk[i].state) != SMC_LINK_UP))
+			continue;
+		confrkeyllc.rt[rt_idx].link_id = lgr->lnk[i].link_id;
+		confrkeyllc.rt[rt_idx].rmb_key =
+			htonl(rmb_rx_elem->mr_rx[i]->rkey);
+		confrkeyllc.rt[rt_idx].rmb_vaddr =
+			cpu_to_be64((u64)rmb_rx_elem->rmb_rx_dma[i]);
+		rt_idx = rt_idx + 1;
+	}
+	confrkeyllc.rt[0].link_id = rt_idx - 1;
+	if (rt_idx == 1) /* nothing to confirm */
+		WARN_ON(1);
+
+	return llc_send_msg(link, (struct llc_msg *)&confrkeyllc,
+			    (enum llc_ctl_ind)lgr->role);
+}
+
+int llc_do_del_rkey(struct smc_link *link, struct rmb_rx_addrs *rmb_rx_elem)
+{
+	struct llc_delete_rkey_msg delrkeyllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+	int prim_lnk_idx;
+
+	memset(&delrkeyllc, 0, sizeof(delrkeyllc));
+	delrkeyllc.hd.type = LLC_DELETE_RKEY;
+	delrkeyllc.hd.length = sizeof(struct llc_delete_rkey_msg);
+	delrkeyllc.hd.flags = 0x00;
+	lgr->llc_ctl[lgr->role].qentry = NULL;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	delrkeyllc.num_rkeys = 1;
+	delrkeyllc.rk[0] =
+		htonl(rmb_rx_elem->mr_rx[prim_lnk_idx]->rkey);
+
+	return llc_send_msg(link, (struct llc_msg *)&delrkeyllc,
+			    (enum llc_ctl_ind)lgr->role);
+}
+
+struct bufs_list_pos {
+	struct list_head	*head;
+	int			ind;
+	struct rmb_rx_addrs	*elem;
+};
+
+static void llc_get_next_rx_rmb_slot(struct smc_link_group *lgr,
+				     struct bufs_list_pos *bufs_pos)
+{
+	int start;
+
+	start = bufs_pos->ind;
+	if (bufs_pos->elem && bufs_pos->elem->list.next !=
+		&bufs_pos->head[start]) {
+		bufs_pos->elem =
+			(struct rmb_rx_addrs *)bufs_pos->elem->list.next;
+		return;
+	}
+	while (start < (SMC_RMBE_SIZE_VALUES - 1)) {
+		start++;
+		if (bufs_pos->head[start].next !=
+			&bufs_pos->head[start]) {
+			bufs_pos->elem = (struct rmb_rx_addrs *)
+						bufs_pos->head[start].next;
+			bufs_pos->ind = start;
+			return;
+		}
+	}
+}
+
+static int llc_add_link_cont(struct smc_link *link, int lnk_idx, u8 link_num,
+			     u8 *num_rkeys_todo, struct bufs_list_pos *pos)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_add_link_cont_msg addc_llcs;
+	int prim_lnk_idx, i;
+	u8 n;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	addc_llcs.link_num = link_num;
+	addc_llcs.num_rkeys = *num_rkeys_todo;
+	n = *num_rkeys_todo;
+	memset(&addc_llcs.reserved2, 0,
+	       sizeof(struct llc_add_link_cont_msg) -
+		      offsetof(struct llc_add_link_cont_msg, reserved2));
+	for (i = 0; i < min_t(u8, n, 2); i++) {
+next_rmb_slot:
+		llc_get_next_rx_rmb_slot(lgr, pos);
+		if (!pos->elem) {
+			addc_llcs.num_rkeys = addc_llcs.num_rkeys -
+							*num_rkeys_todo;
+			*num_rkeys_todo = 0;
+			break;
+		}
+		if (!pos->elem->mr_rx[prim_lnk_idx])
+			goto next_rmb_slot;
+		addc_llcs.rt[i].rmb_key =
+			htonl(pos->elem->mr_rx[prim_lnk_idx]->rkey);
+		addc_llcs.rt[i].rmb_key_new =
+			htonl(pos->elem->mr_rx[lnk_idx]->rkey);
+		addc_llcs.rt[i].rmb_vaddr_new =
+			cpu_to_be64((u64)pos->elem->rmb_rx_dma[lnk_idx]);
+		(*num_rkeys_todo)--;
+	}
+	addc_llcs.hd.type = LLC_ADD_LINK_CONT;
+	addc_llcs.hd.length = sizeof(struct llc_msg);
+	addc_llcs.hd.version = SMC_CLC_V1;
+	if (lgr->role == SMC_SERV)
+		addc_llcs.hd.flags = 0x00;
+	else
+		addc_llcs.hd.flags = 0x80;
+	llc_send_msg(link, (struct llc_msg *)&addc_llcs, LLC_SERV);
+	return 0;
+}
+
+int llc_send_test_link(struct smc_link *link, u8 udata[])
+{
+	struct llc_test_link_msg testllc;
+	struct smc_link_group *lgr = link->lnk_grp;
+
+	testllc.hd.type = LLC_TEST_LINK;
+	memset(&testllc.user_data, 0, sizeof(testllc.user_data));
+	strncpy(testllc.user_data, udata, sizeof(testllc.user_data));
+	testllc.hd.length = sizeof(struct llc_test_link_msg);
+	testllc.hd.flags = 0x00;
+	lgr->llc_ctl[LLC_TESTLINK].qentry = NULL;
+	testllc.hd.version = SMC_CLC_V1;
+	return llc_send_msg(link, (struct llc_msg *)&testllc, LLC_TESTLINK);
+}
+
+int llc_get_fail_cause(struct smc_link_group *lgr, enum llc_ctl_ind initiator,
+		       struct smc_sock *smc)
+{
+	struct smc_decline_clc_msg dclc;
+	int reason_code, rc;
+
+	if ((initiator < 3) &&
+	    ((lgr->llc_ctl[initiator].ctrl_elem != LLC_SEND_POSTED) ||
+	     !lgr->llc_ctl[initiator].wc_status)) {
+		/* llc send failure */
+		return -ECOMM;
+	}
+	if (!smc)
+		return -ETIMEDOUT;
+	/* clc decl received or llc timeout */
+	memset(&dclc, 0, sizeof(dclc));
+	rc = clc_wait_msg(smc, SMC_CLC_NONBLOCK, (char *)&dclc,
+			  sizeof(struct smc_decline_clc_msg),
+			  SMC_CLC_DECLINE, &reason_code);
+	/* clc decline received */
+	return rc;
+}
+
+static u8 llc_count_rx_rmb_slot(struct smc_link_group *lgr, int lnk_idx)
+{
+	struct rmb_rx_addrs *entry_rx;
+	int i;
+	u8 count;
+
+	count = 0;
+	for (i = 0; i < 16; i++) {
+		list_for_each_entry(entry_rx, &lgr->rmb_rx_bufs[i], list) {
+			if (entry_rx->mr_rx[lnk_idx])
+				count++;
+		}
+	}
+	return count;
+}
+
+int smc_find_rtoken_by_link(struct smc_link_group *lgr, int lnk_idx, u32 rkey)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		if ((lgr->rtok[i].rkey[lnk_idx] == rkey) &&
+		    (lgr->rtok[i].link_id[lnk_idx] ==
+						lgr->lnk[lnk_idx].link_id) &&
+		    (atomic_read(&lgr->rtok[i].active) == 1))
+			return i;
+	}
+	return -ENOENT;
+}
+
+static void llc_set_rtoken(struct smc_link_group *lgr, int link_n1, int link_n2,
+			   struct rmb_rtoken2 *rtoken)
+{
+	u32 rkey;
+	int rtok_idx;
+
+	rkey = ntohl(rtoken->rmb_key);
+	rtok_idx = smc_find_rtoken_by_link(lgr, link_n1, rkey);
+	if (rtok_idx == -ENOENT)
+		return;
+	lgr->rtok[rtok_idx].rkey[link_n2] = ntohl(rtoken->rmb_key_new);
+	lgr->rtok[rtok_idx].vaddr[link_n2] = be64_to_cpu(rtoken->rmb_vaddr_new);
+	lgr->rtok[rtok_idx].link_id[link_n2] = lgr->lnk[link_n2].link_id;
+}
+
+int smc_get_rtoken(struct smc_link_group *lgr)
+{
+	int i;
+
+	for (i = 0; i < SMC_MAX_RMB; i++) {
+		if (atomic_cmpxchg(&lgr->rtok[i].active, 0, 1) == 0)
+			return i;
+	}
+	return -ENOSPC;
+}
+
+static void llc_set_rtoken_by_link(struct smc_link_group *lgr, int link_n1,
+				   int rtoken_idx, struct rmb_rtoken3 *rtoken)
+{
+	int lnk_idx, found = 0;
+
+	for (lnk_idx = 0; lnk_idx <= SMC_MAX_SYM_LINKS; lnk_idx++) {
+		if (lgr->lnk[lnk_idx].link_id == link_n1) {
+			found = 1;
+			break;
+		}
+	}
+	if (!found) {
+		WARN_ON(1);
+		return;
+	}
+	lgr->rtok[rtoken_idx].rkey[lnk_idx] = ntohl(rtoken->rmb_key);
+	lgr->rtok[rtoken_idx].vaddr[lnk_idx] = be64_to_cpu(rtoken->rmb_vaddr);
+	lgr->rtok[rtoken_idx].link_id[lnk_idx] = link_n1;
+}
+
+static int smc_clear_rtokens_by_link(struct smc_link_group *lgr, int link_n1,
+				     __be32 rmb_rkey)
+{
+	int i, lnk_idx, rtoken_idx, found = 0;
+	u32 rkey;
+
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id == link_n1) {
+			found = 1;
+			break;
+		}
+	}
+	if (!found)
+		return -ENODEV;
+	lnk_idx = i;
+	rkey = ntohl(rmb_rkey);
+	rtoken_idx = smc_find_rtoken_by_link(lgr, lnk_idx, rkey);
+	found = (rtoken_idx < 0 ? 0 : 1);
+	if (!found)
+		return -ENODEV;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		lgr->rtok[rtoken_idx].rkey[i] = 0;
+		lgr->rtok[rtoken_idx].vaddr[i] = 0;
+		lgr->rtok[rtoken_idx].link_id[i] = 0;
+	}
+	atomic_set(&lgr->rtok[rtoken_idx].active, 0);
+	return 0;
+}
+
+static int llc_alloc_alt_link(struct smc_link_group *lgr,
+			      enum lgr_type lgr_new_t)
+{
+	int i, lnk_idx, total_links = 0;
+
+	if ((lgr->lgr_type != SINGLE) &&
+	    ((lgr_new_t == ASYMMETRIC_LOCAL) ||
+	     (lgr_new_t == ASYMMETRIC_PEER)))
+		return -EMLINK;
+	if ((lgr_new_t == ASYMMETRIC_LOCAL) ||
+	    (lgr_new_t == ASYMMETRIC_PEER)) {
+		lnk_idx = SMC_MAX_SYM_LINKS;
+		for (i = SMC_MAX_SYM_LINKS; i > 0; i--) {
+			if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_FREED) {
+				lnk_idx = i;
+				break;
+			}
+		}
+	} else {
+		lnk_idx = SMC_MAX_SYM_LINKS + 1;
+		total_links = 0;
+		for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+			if (atomic_read(&lgr->lnk[i].state) == SMC_LINK_FREED) {
+				if (lnk_idx > SMC_MAX_SYM_LINKS)
+					lnk_idx = i;
+			} else {
+				total_links++;
+			}
+		}
+		if ((lgr->lgr_type == SYMMETRIC) &&
+		    (total_links >= SMC_MAX_SYM_LINKS))
+			return -EMLINK;
+		if (lnk_idx > SMC_MAX_SYM_LINKS)
+			return -EMLINK;
+	}
+	return lnk_idx;
+}
+
+static void llc_deny_req(struct llc_qentry *qentry, u8 flag, u8 retc,
+			 enum llc_ctl_ind ctl_idx)
+{
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	qentry->msg.hd.flags |= flag;
+	qentry->msg.hd.version |= retc;
+	llc_send_msg(qentry->link, &qentry->msg, ctl_idx);
+	kfree(qentry);
+}
+
+static void llc_save_add_link_info(struct smc_link *link,
+				   struct llc_add_link_msg *add_llc)
+{
+	memcpy(link->gid_peer, ((union ib_gid *)add_llc->sender_gid)->raw, 16);
+	memcpy(link->mac_peer, add_llc->sender_mac, 6);
+	ntoh_three(&link->qp_num_peer, add_llc->sender_qp_num);
+	ntoh_three(&link->psn_peer, add_llc->initial_psn);
+	link->mtu_peer = add_llc->flags2 & 0x0f;
+}
+
+static int llc_cli_rkey_exchange(struct smc_link *link,
+				 struct smc_link *link_new,
+				 struct smc_sock *smc, int lnk_idx)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_cont_msg *addc_llc;
+	struct bufs_list_pos pos = {
+		.head = lgr->rmb_rx_bufs,
+		.ind = -1,
+		.elem = NULL,
+	};
+	int prim_lnk_idx, i;
+	u8 n, num_rkeys_send, num_rkeys_recv;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	num_rkeys_send = llc_count_rx_rmb_slot(lgr, prim_lnk_idx);
+	do {
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK_CONT);
+		if (!qentry) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return llc_get_fail_cause(lgr, LLC_SERV, smc);
+		}
+		addc_llc = (struct llc_add_link_cont_msg *)&qentry->msg;
+		num_rkeys_recv = addc_llc->num_rkeys;
+		n = addc_llc->num_rkeys;
+		for (i = 0; i < min_t(u8, n, 2); i++) {
+			llc_set_rtoken(lgr, prim_lnk_idx, lnk_idx,
+				       &addc_llc->rt[i]);
+			num_rkeys_recv--;
+		}
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		kfree(qentry);
+		llc_add_link_cont(link, lnk_idx, link_new->link_id,
+				  &num_rkeys_send, &pos);
+	} while (num_rkeys_send || num_rkeys_recv);
+	return 0;
+}
+
+static int llc_cli_conf_link(struct smc_link *link,
+			     struct smc_roce_defs *rocedef,
+			     struct smc_link *link_new, int lnk_idx,
+			     enum lgr_type lgr_new_t)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_del_link_msg *del_llc;
+	int rc = 0, cas;
+
+	/* receive CONFIRM LINK request over RoCE fabric */
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO, 0);
+	if (!qentry) {
+		rc = llc_send_del_link(link, link_new, 0,
+				       LLC_DEL_LOST_PATH,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	if (qentry->msg.hd.type == LLC_CONFIRM_LINK) {
+		rc = smc_modify_qp_rts(link_new);
+		cas = 0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+		if (rc || !(qentry->msg.hd.flags & 0xf) ||
+		    !lgr->cache_align_size || (lgr->cache_align_size != cas)) {
+			/* no cache alignment specified, decline */
+			kfree(qentry);
+			rc = llc_send_del_link(link, link_new, 0,
+					       LLC_DEL_PROT_VIOL,
+					       LLC_NO_NOTIFY);
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return -ENOLINK;
+		}
+		if (!lgr->cache_align_size)
+			lgr->cache_align_size = cas;
+		smc_check_qp_attr(link_new);
+		/* send CONFIRM LINK response over RoCE fabric */
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		rc = llc_send_confirm_link(link_new,
+					   (u8 *)&rocedef->mac,
+					   &rocedef->gid, LLC_RESP,
+					   LLC_NO_NOTIFY);
+		kfree(qentry);
+		if (rc) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			rc = llc_send_del_link(link, link_new, 0,
+					       LLC_DEL_LOST_PATH,
+					       LLC_NO_NOTIFY);
+			if (rc && smc_lnk_downing(&link->state))
+				smc_link_down(lgr, link - &lgr->lnk[0]);
+			return -ENOLINK;
+		}
+		atomic_set(&link_new->state, SMC_LINK_UP);
+		lgr->lgr_type = lgr_new_t;
+	} else {
+		del_llc = (struct llc_del_link_msg *)&qentry->msg;
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		qentry->msg.hd.flags |= LLC_FLAG_RESP;
+		llc_send_msg(link, &qentry->msg, LLC_SERV);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		kfree(qentry);
+		return -ENOLINK;
+	}
+	return 0;
+}
+
+int llc_cli_add_link(struct smc_link *link, struct llc_qentry *qentry,
+		     struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_link *link_new;
+	struct llc_add_link_msg *add_llc;
+	struct smc_roce_defs rocedef;
+	enum lgr_type lgr_new_t = SYMMETRIC;
+	int lnk_idx, rc;
+	int reuse_lnk_idx = SMC_MAX_SYM_LINKS + 1;
+
+	rocedef.ibdev = NULL;
+	smc_find_alt_roce_resources(&rocedef, lgr, link);
+	add_llc = (struct llc_add_link_msg *)&qentry->msg;
+	if (!memcmp(add_llc->sender_gid, link->gid_peer, 16) &&
+	    !memcmp(add_llc->sender_mac, link->mac_peer, 6)) {
+		if (!rocedef.ibdev) {
+			llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+				     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+			return 0;
+		}
+		lgr_new_t = ASYMMETRIC_PEER;
+	}
+	if (rocedef.ibdev) {
+		if (!smc_port_active(rocedef.ibdev, rocedef.port)) {
+			llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+				     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+			return 0;
+		}
+	} else {
+		lgr_new_t = ASYMMETRIC_LOCAL;
+		memcpy(&rocedef, &link->roce_defs,
+		       sizeof(struct smc_roce_defs));
+		reuse_lnk_idx = link - &lgr->lnk[0];
+	}
+	lnk_idx = llc_alloc_alt_link(lgr, lgr_new_t);
+	if (lnk_idx < 0) {
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		return 0;
+	}
+	rc = smc_create_link(lgr, lnk_idx, &rocedef);
+	if (rc) {
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		return 0;
+	}
+	link_new = &lgr->lnk[lnk_idx];
+	llc_save_add_link_info(link_new, add_llc);
+	rc = smc_ready_link(link_new);
+	if (rc) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		llc_deny_req(qentry, LLC_FLAG_ADD_LNK_REJ,
+			     LLC_FLAG_NO_ALT_PATH, LLC_NO_NOTIFY);
+		smc_free_link(lgr, lnk_idx);
+		return 0;
+	}
+	link_new->link_id = add_llc->link_num;
+	kfree(qentry);
+	smc_map_rmbs_to_link(lgr, lnk_idx, reuse_lnk_idx);
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	llc_send_add_link(link, &rocedef, link_new, LLC_RESP, LLC_SERV);
+	rc = llc_cli_rkey_exchange(link, link_new, smc, lnk_idx);
+	if (!rc)
+		rc = llc_cli_conf_link(link, &rocedef, link_new, lnk_idx,
+				       lgr_new_t);
+	return rc;
+}
+
+static int llc_srv_rkey_exchange(struct smc_link *link,
+				 struct smc_link *link_new,
+				 struct smc_sock *smc,
+				 int lnk_idx)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_cont_msg *addc_llc;
+	struct bufs_list_pos pos = {
+		.head = lgr->rmb_rx_bufs,
+		.ind = -1,
+		.elem = NULL,
+	};
+	int prim_lnk_idx;
+	int i;
+	u8 n, num_rkeys_send, num_rkeys_recv;
+
+	prim_lnk_idx = link - &lgr->lnk[0];
+	num_rkeys_send = llc_count_rx_rmb_slot(lgr, prim_lnk_idx);
+	do {
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		llc_add_link_cont(link, lnk_idx, link_new->link_id,
+				  &num_rkeys_send, &pos);
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK_CONT);
+		if (!qentry) {
+			atomic_set(&link_new->state, SMC_LINK_DOWN);
+			smc_free_link(lgr, lnk_idx);
+			return llc_get_fail_cause(lgr, LLC_SERV, smc);
+		}
+		addc_llc = (struct llc_add_link_cont_msg *)&qentry->msg;
+		num_rkeys_recv = addc_llc->num_rkeys;
+		n = addc_llc->num_rkeys;
+		for (i = 0; i < min_t(u8, n, 2); i++) {
+			llc_set_rtoken(lgr, prim_lnk_idx, lnk_idx,
+				       &addc_llc->rt[i]);
+			num_rkeys_recv--;
+		}
+		kfree(qentry);
+	} while (num_rkeys_send || num_rkeys_recv);
+	return 0;
+}
+
+static int llc_srv_conf_link(struct smc_link *link,
+			     struct smc_roce_defs *rocedef,
+			     struct smc_link *link_new, int lnk_idx,
+			     enum lgr_type lgr_new_t)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct llc_qentry *qentry = NULL;
+	int cas, rc;
+
+	/* send CONFIRM LINK request over the RoCE fabric */
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	rc = llc_send_confirm_link(link_new, (u8 *)&rocedef->mac,
+				   &rocedef->gid, LLC_REQ, LLC_SERV);
+	/* receive CONFIRM LINK response over the RoCE fabric */
+	if (!rc)
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_CONFIRM_LINK);
+	if (rc || !qentry) {
+		/* send DELETE LINK */
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		rc = llc_send_del_link(link, link_new, 0, LLC_DEL_LOST_PATH,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		kfree(qentry);
+		if (rc) {
+			smc_terminate_conn(lgr);
+			return rc;
+		}
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	cas = 0x00000001 << (((int)(qentry->msg.hd.flags & 0x0f)) + 5);
+	if (!(qentry->msg.hd.flags & 0xf) ||
+	    !lgr->cache_align_size || (lgr->cache_align_size != cas)) {
+		/* no cache alignment specified, decline */
+		kfree(qentry);
+		rc = llc_send_del_link(link, link_new, 0,
+				       LLC_DEL_PROT_VIOL,
+				       LLC_NO_NOTIFY);
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		if (rc) {
+			smc_terminate_conn(lgr);
+			return rc;
+		}
+		smc_free_link(lgr, lnk_idx);
+		return -ENOLINK;
+	}
+	if (!lgr->cache_align_size)
+		lgr->cache_align_size = cas;
+	atomic_set(&link_new->state, SMC_LINK_UP);
+	lgr->lgr_type = lgr_new_t;
+	kfree(qentry);
+	return 0;
+}
+
+int llc_srv_add_link(struct smc_link *link, struct smc_roce_defs *rocedef,
+		     struct smc_sock *smc)
+{
+	struct smc_link_group *lgr = link->lnk_grp;
+	struct smc_link *link_new;
+	struct llc_qentry *qentry = NULL;
+	struct llc_add_link_msg *add_llc;
+	enum lgr_type lgr_new_t = SYMMETRIC;
+	int lnk_idx, rc = 0;
+	int reuse_lnk_idx = SMC_MAX_SYM_LINKS + 1;
+
+	if (rocedef->ibdev) {
+		if (!smc_port_active(rocedef->ibdev, rocedef->port))
+			return -ENOLINK;
+	} else {
+		lgr_new_t = ASYMMETRIC_LOCAL;
+		memcpy(rocedef, &link->roce_defs, sizeof(struct smc_roce_defs));
+		reuse_lnk_idx = link - &lgr->lnk[0];
+	}
+	lnk_idx = llc_alloc_alt_link(lgr, lgr_new_t);
+	if (lnk_idx < 0)
+		return 0;
+	rc = smc_create_link(lgr, lnk_idx, rocedef);
+	if (rc)
+		return rc;
+	link_new = &lgr->lnk[lnk_idx];
+	rc = llc_send_add_link(link, rocedef, link_new, LLC_REQ, LLC_SERV);
+	if (!rc) {
+		/* receive ADD LINK response over the RoCE fabric */
+		qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_FIRST_TIMEO,
+				  LLC_ADD_LINK);
+	}
+	if (rc || !qentry) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		return llc_get_fail_cause(lgr, LLC_SERV, smc);
+	}
+	add_llc = (struct llc_add_link_msg *)&qentry->msg;
+	if (add_llc->hd.flags & LLC_FLAG_ADD_LNK_REJ) {
+		atomic_set(&link_new->state, SMC_LINK_DOWN);
+		smc_free_link(lgr, lnk_idx);
+		kfree(qentry);
+		return -ENOLINK;
+	}
+	if ((lgr->lgr_type == SINGLE) &&
+	    (!memcmp(add_llc->sender_gid, link->gid_peer, 16) &&
+	    !memcmp(add_llc->sender_mac, link->mac_peer, 6)))
+		lgr_new_t = ASYMMETRIC_PEER;
+	llc_save_add_link_info(link_new, add_llc);
+	kfree(qentry);
+	smc_ready_link(link_new);
+	smc_map_rmbs_to_link(lgr, lnk_idx, reuse_lnk_idx);
+	llc_srv_rkey_exchange(link, link_new, smc, lnk_idx);
+	rc = llc_srv_conf_link(link, rocedef, link_new, lnk_idx, lgr_new_t);
+	return rc;
+}
+
+static bool start_llc_agent(enum smc_role initiator,
+			    struct llc_qentry *qentry)
+{
+	struct smc_link_group *lgr = qentry->link->lnk_grp;
+
+	spin_lock_bh(&lgr->llc_ctl_lock);
+	if (lgr->llc_ctl[initiator].active) {
+		if (((qentry->msg.hd.type == LLC_ADD_LINK) ||
+		     (qentry->msg.hd.type == LLC_DEL_LINK)) &&
+		    !lgr->delayed_q) {
+			lgr->delayed_q = qentry;
+		} else {
+			kfree(qentry);
+		}
+		spin_unlock_bh(&lgr->llc_ctl_lock);
+		return 0;
+	}
+	lgr->llc_ctl[initiator].active = llc_msg_to_grp(qentry->msg.hd.type);
+	if (qentry == lgr->delayed_q)
+		lgr->delayed_q = NULL;
+	spin_unlock_bh(&lgr->llc_ctl_lock);
+	lgr->llc_ctl[initiator].qentry = qentry;
+	return 1;
+}
+
+void llc_cancel_wait(struct smc_link *lnk)
+{
+	struct smc_link_group *lgr = lnk->lnk_grp;
+	int i;
+	bool cancelled = false;
+
+	for_each_set_bit(i, lnk->pending_bit_mask, lnk->send_wr_num) {
+		u8 ctl_idx = lnk->pending[i].post_ctl;
+
+		if (!lnk->pending[i].conn &&
+		    (ctl_idx != LLC_NO_NOTIFY) &&
+		    lgr->llc_ctl[ctl_idx].active &&
+		    !lgr->llc_ctl[ctl_idx].qentry) {
+			lgr->llc_ctl[ctl_idx].qentry = ERR_PTR(-EPIPE);
+			cancelled = true;
+		}
+	}
+	if (cancelled)
+		wake_up_interruptible(&lgr->llc_waiter);
+}
+
+void llc_kill_waiters(struct smc_link_group *lgr)
+{
+	int i;
+	bool cancelled = false;
+
+	for (i = 0; i < LLC_NO_NOTIFY; i++) {
+		if (lgr->llc_ctl[i].active && !lgr->llc_ctl[i].qentry) {
+			lgr->llc_ctl[i].qentry = ERR_PTR(-EPIPE);
+			cancelled = true;
+		}
+	}
+	if (cancelled)
+		wake_up_interruptible(&lgr->llc_waiter);
+}
+
+static void llc_process_cli_add_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct smc_link *lnk;
+
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	mutex_lock(&lgr->conf_mutex);
+	llc_cli_add_link(lnk, qentry, NULL);
+	mutex_unlock(&lgr->conf_mutex);
+}
+
+static void llc_del_asym_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc_resp;
+	struct smc_link *link = &lgr->lnk[0];
+	int asym_lnk = SMC_MAX_SYM_LINKS + 1;
+	int i, j, k, rc;
+
+	/* determine asymmetric link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		for (j = i + 1; j <= SMC_MAX_SYM_LINKS; j++) {
+			if (!memcmp(lgr->lnk[i].roce_defs.gid.raw,
+				    lgr->lnk[j].roce_defs.gid.raw, 16)) {
+				asym_lnk = j; /* asym_lnk is i or j */
+				break;
+			}
+		}
+		if (asym_lnk < SMC_MAX_SYM_LINKS + 1)
+			break;
+	}
+	if (asym_lnk == SMC_MAX_SYM_LINKS + 1)
+		return; /* no asymmetric link */
+	for (k = 0; k <= SMC_MAX_SYM_LINKS; k++) {
+		if ((k != i) &&
+		    !memcmp(lgr->lnk[i].gid_peer, lgr->lnk[k].gid_peer, 16)) {
+			asym_lnk = i;
+			break;
+		}
+		if ((k != j) &&
+		    !memcmp(lgr->lnk[j].gid_peer, lgr->lnk[k].gid_peer, 16))
+			break;
+	}
+	if (k == SMC_MAX_SYM_LINKS + 1)
+		return; /* no asymmetric link */
+	if (asym_lnk == 0)
+		link = &lgr->lnk[1];
+	lgr->llc_ctl[LLC_SERV].active = LLC_GRP_DEL_LINK;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (!smc_lnk_downing(&lgr->lnk[asym_lnk].state))
+		return;
+	rc = smc_switch_conns(lgr, asym_lnk, 0);
+	smc_wait_no_pending_sends_on_link(&lgr->lnk[asym_lnk]);
+	if (rc == -ENOENT) {
+		smc_free_link(lgr, asym_lnk);
+		goto out;
+	}
+	rc = llc_send_del_link(link, &lgr->lnk[asym_lnk],
+			       LLC_FLAG_DEL_ORDERLY, LLC_DEL_NO_ASYM_NEEDED,
+			       LLC_SERV);
+	if (rc) {
+		if (smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+		goto out;
+	}
+	qentry = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO, LLC_DEL_LINK);
+	if (!qentry) {
+		if (smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+	} else {
+		del_llc_resp = (struct llc_del_link_msg *)&qentry->msg;
+		if ((ntohl(del_llc_resp->reason) != LLC_DEL_NO_ASYM_NEEDED) &&
+		    smc_lnk_downing(&link->state))
+			smc_link_down(lgr, link - &lgr->lnk[0]);
+		atomic_set(&lgr->lnk[asym_lnk].state, SMC_LINK_DOWN);
+	}
+	kfree(qentry);
+	smc_free_link(lgr, asym_lnk);
+out:
+	return;
+}
+
+static void llc_process_srv_add_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_add_link_fake_msg *add_llc;
+	struct smc_link *lnk;
+	struct smc_roce_defs rocdefs;
+	int rc;
+	enum lgr_type old_lgr_type;
+
+	rocdefs.ibdev = NULL;
+	rocdefs.port = 0;
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	add_llc = (struct llc_add_link_fake_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (qentry->msg.hd.flags & LLC_FLAG_LOCAL) {
+		rocdefs.ibdev = add_llc->ibdev;
+		rocdefs.port = add_llc->port_num;
+		rocdefs.vlan = lgr->vlan;
+		rc = smc_gid_by_dev(&rocdefs);
+		if (rc) {
+			kfree(qentry);
+			wake_up_interruptible(&lgr->llc_waiter);
+			return;
+		}
+	} else {
+		smc_find_alt_roce_resources(&rocdefs, lgr, lnk);
+	}
+	kfree(qentry);
+	qentry = NULL;
+	old_lgr_type = lgr->lgr_type;
+	mutex_lock(&lgr->conf_mutex);
+	rc = llc_srv_add_link(lnk, &rocdefs, NULL);
+	if (!rc && (lgr->lgr_type == SYMMETRIC)) {
+		/* delete the superfluos asymmetric link now */
+		llc_del_asym_link(lgr);
+	}
+	mutex_unlock(&lgr->conf_mutex);
+}
+
+void llc_process_add_link(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_add_link_work);
+
+	if (list_empty(&lgr->list) || (lgr->lgr_type == NONE)) {
+		/* link group to be deleted */
+		kfree(lgr->llc_ctl[LLC_SERV].qentry);
+		wake_up_interruptible(&lgr->llc_waiter);
+		goto out;
+	}
+	if (lgr->role == SMC_CLNT)
+		llc_process_cli_add_link(lgr);
+	else
+		llc_process_srv_add_link(lgr);
+	llc_stop(lgr, LLC_GRP_ADD_LINK);
+out:
+	return;
+}
+
+static void llc_process_cli_del_all(struct smc_link_group *lgr,
+				    struct smc_link *link,
+				    struct llc_qentry *qentry)
+{
+	struct llc_del_link_msg *del_llc;
+	int empty;
+
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	del_llc->reason = 0;
+	mutex_lock(&lgr->conf_mutex);
+	llc_send_msg(link, &qentry->msg, LLC_SERV); /* send response */
+	memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+	mutex_unlock(&lgr->conf_mutex);
+	wake_up_interruptible(&lgr->llc_waiter);
+	/* delete link group at all */
+	spin_lock_bh(&smc_lgr_list.lock);
+	if (!list_empty(&lgr->list)) {
+		list_move_tail(&lgr->list, &smc_lgr_freeing_list);
+		spin_unlock_bh(&smc_lgr_list.lock);
+		mutex_lock(&lgr->conf_mutex);
+		read_lock_bh(&lgr->conns_lock);
+		empty = RB_EMPTY_ROOT(&lgr->conns_all);
+		read_unlock_bh(&lgr->conns_lock);
+		mutex_unlock(&lgr->conf_mutex);
+		if (!empty)
+			smc_terminate_conn(lgr);
+		llc_kill_waiters(lgr);
+		cancel_work_sync(&lgr->llc_event_work);
+		smc_wait_no_pending_sends_on_link(link);
+		schedule_delayed_work(&lgr->free_lgr_final_work,
+				      2 * TCP_TIMEWAIT_LEN);
+	} else {
+		spin_unlock_bh(&smc_lgr_list.lock);
+	}
+}
+
+static void llc_process_cli_del_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc;
+	struct smc_link *lnk;
+	int i, link_found;
+
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	del_llc->hd.flags |= LLC_FLAG_RESP;
+	if (del_llc->hd.flags & LLC_FLAG_DEL_ALL) {
+		llc_process_cli_del_all(lgr, lnk, qentry);
+		goto out;
+	}
+	mutex_lock(&lgr->conf_mutex);
+	/* delete single link */
+	link_found = 0;
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id != del_llc->link_num)
+			continue;
+		link_found = 1;
+		if (lnk->link_id == del_llc->link_num)
+			lgr->lgr_type = NONE; /* delete link of received llc */
+		else
+			llc_send_msg(lnk, &qentry->msg, LLC_SERV); /* response*/
+		if (smc_lnk_downing(&lgr->lnk[i].state)) {
+			smc_switch_conns(lgr, i, 0);
+			smc_wait_no_pending_sends_on_link(&lgr->lnk[i]);
+		}
+		if (ntohl(del_llc->reason) != LLC_DEL_NO_ASYM_NEEDED) {
+			lgr->lgr_type = SINGLE;
+			del_llc->reason = 0;
+		}
+		memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+		smc_free_link(lgr, i);
+		mutex_unlock(&lgr->conf_mutex);
+		wake_up_interruptible(&lgr->llc_waiter);
+		break;
+	}
+	if (!link_found) {
+		del_llc->reason = htonl(SMC_LLC_DEL_NOLNK);
+		llc_send_msg(lnk, &qentry->msg, LLC_SERV);
+		memset(&lgr->llc_ctl[LLC_SERV], 0, sizeof(struct llc_ctl));
+		mutex_unlock(&lgr->conf_mutex);
+		wake_up_interruptible(&lgr->llc_waiter);
+	}
+out:
+	kfree(qentry);
+}
+
+static void llc_process_srv_del_link(struct smc_link_group *lgr)
+{
+	struct llc_qentry *qentry;
+	struct llc_del_link_msg *del_llc;
+	struct smc_link *lnk;
+	struct smc_roce_defs rocdefs;
+	int i, lnk_idx = -1, found = 0;
+
+	mutex_lock(&lgr->conf_mutex);
+	qentry = lgr->llc_ctl[LLC_SERV].qentry;
+	lnk = qentry->link;
+	del_llc = (struct llc_del_link_msg *)&qentry->msg;
+	lgr->llc_ctl[LLC_SERV].qentry = NULL;
+	if (qentry->msg.hd.flags & LLC_FLAG_DEL_ALL) {
+		/* delete entire lgr */
+		goto out;
+	}
+	/* delete single link */
+	for (i = 0; i <= SMC_MAX_SYM_LINKS; i++) {
+		if (lgr->lnk[i].link_id == del_llc->link_num) {
+			found = 1;
+			lnk_idx = i;
+		}
+	}
+	lgr->lgr_type = SINGLE;
+	if (found && smc_lnk_downing(&lgr->lnk[lnk_idx].state)) {
+		smc_switch_conns(lgr, lnk_idx, 0);
+		smc_wait_no_pending_sends_on_link(&lgr->lnk[lnk_idx]);
+	}
+	if ((lgr->lgr_type != NONE) &&
+	    (found || (qentry->msg.hd.type == 0x80 + LLC_DEL_LINK))) {
+		struct llc_qentry *qentry2;
+
+		lgr->llc_ctl[LLC_SERV].qentry = NULL;
+		qentry->msg.hd.type = LLC_DEL_LINK;
+		qentry->msg.hd.flags &= ~LLC_FLAG_LOCAL;
+		llc_send_msg(lnk, &qentry->msg, LLC_SERV);
+		qentry2 = llc_wait(lgr, LLC_SERV, LLC_WAIT_TIMEO,
+				   LLC_DEL_LINK);
+		kfree(qentry2);
+	}
+	if (found) {
+		smc_free_link(lgr, lnk_idx);
+		if (qentry && (lgr->lgr_type == SINGLE) &&
+		    !list_empty(&lgr->list)) {
+			/* setup asymm alt link */
+			rocdefs.ibdev = NULL;
+			smc_find_alt_roce_resources(&rocdefs, lgr, lnk);
+			if (!rocdefs.ibdev) {
+				lgr->llc_ctl[LLC_SERV].active =
+							LLC_GRP_ADD_LINK;
+				llc_srv_add_link(lnk, &rocdefs, NULL);
+			}
+		}
+	}
+out:
+	mutex_unlock(&lgr->conf_mutex);
+	kfree(qentry);
+	llc_stop(lgr, LLC_GRP_DEL_LINK);
+}
+
+void llc_process_del_link(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_del_link_work);
+
+	if (lgr->role == SMC_CLNT)
+		llc_process_cli_del_link(lgr);
+	else
+		llc_process_srv_del_link(lgr);
+}
+
+void llc_process_confirm_rkey(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+					    llc_conf_rkey_work);
+	struct smc_link *lnk;
+	struct llc_qentry *qentry;
+	struct llc_confirm_rkey_msg *rk_llc;
+	int rt_idx = 0, i, rc = 0;
+	enum llc_ctl_ind initiator;
+	u8 num_entries, more = 0;
+
+	initiator = (lgr->role == SMC_SERV) ? LLC_CLNT : LLC_SERV;
+	qentry = lgr->llc_ctl[initiator].qentry;
+	lnk = qentry->link;
+	rk_llc = (struct llc_confirm_rkey_msg *)&qentry->msg;
+	lgr->llc_ctl[initiator].qentry = NULL;
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	if (lgr->llc_ctl[LLC_SERV].active > LLC_GRP_CONF_RKEY) {
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_NEG;
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_RETRY;
+	} else {
+		num_entries = rk_llc->rt[0].link_id;
+		rt_idx = smc_get_rtoken(lgr);
+		llc_set_rtoken_by_link(lgr, lnk->link_id, rt_idx,
+				       &rk_llc->rt[0]);
+		for (i = 1; i <= min_t(u8, num_entries, 2); i++)
+			llc_set_rtoken_by_link(lgr, rk_llc->rt[i].link_id,
+					       rt_idx, &rk_llc->rt[i]);
+		more = (num_entries > 2) ? 1 : 0;
+	}
+	rc = llc_send_msg(lnk, &qentry->msg, LLC_NO_NOTIFY);
+	kfree(qentry);
+again:
+	if (more && (rc >= 0)) {
+		qentry = llc_wait(lgr, (enum smc_role)initiator, LLC_WAIT_TIMEO,
+				  LLC_CONFIRM_RKEY_CONT);
+		if (!qentry) {
+			more = 0;
+		} else {
+			struct llc_conf_rkey_cont_msg *rkc_llc =
+				(struct llc_conf_rkey_cont_msg *)&qentry->msg;
+			lgr->llc_ctl[initiator].qentry = NULL;
+			qentry->msg.hd.flags |= LLC_FLAG_RESP;
+			num_entries = rkc_llc->num_rkeys;
+			for (i = 0; i <= min_t(u8, num_entries, 2); i++) {
+				llc_set_rtoken_by_link(lgr,
+						       rkc_llc->rt[i].link_id,
+						       rt_idx, &rkc_llc->rt[i]);
+			}
+			more = (num_entries > 3) ? 1 : 0;
+			rc = llc_send_msg(lnk, &qentry->msg, LLC_NO_NOTIFY);
+			kfree(qentry);
+		}
+		goto again;
+	}
+	memset(&lgr->llc_ctl[initiator], 0, sizeof(struct llc_ctl));
+}
+
+void llc_process_delete_rkey(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_del_rkey_work);
+	struct smc_link *lnk;
+	struct llc_qentry *qentry;
+	struct llc_delete_rkey_msg *drk_llc;
+	int initiator;
+
+	initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+	qentry = lgr->llc_ctl[initiator].qentry;
+	lnk = qentry->link;
+	drk_llc = (struct llc_delete_rkey_msg *)&qentry->msg;
+	lgr->llc_ctl[initiator].qentry = NULL;
+	qentry->msg.hd.flags |= LLC_FLAG_RESP;
+	if (lgr->llc_ctl[LLC_SERV].active > LLC_GRP_CONF_RKEY) {
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_NEG;
+		qentry->msg.hd.flags |= LLC_FLAG_RKEY_RETRY;
+	} else {
+		int i, rc;
+		u8 local_mask = 0, err_val;
+
+		err_val = 0x80;
+		for (i = 0; i < min_t(u8, drk_llc->num_rkeys, 8); i++) {
+			rc = smc_clear_rtokens_by_link(lgr, lnk->link_id,
+						       drk_llc->rk[i]);
+			if (rc < 0)
+				local_mask = local_mask + err_val;
+			err_val = err_val / 2;
+		}
+		if (local_mask) {
+			drk_llc->hd.flags |= LLC_FLAG_RKEY_NEG;
+			drk_llc->err_mask = local_mask;
+		}
+	}
+	llc_send_msg(lnk, &qentry->msg, initiator);
+	kfree(qentry);
+	memset(&lgr->llc_ctl[initiator], 0, sizeof(struct llc_ctl));
+}
+
+static void llc_handle_event(struct llc_qentry *qentry,
+			     struct smc_link_group *lgr)
+{
+	struct smc_link *lnk = qentry->link;
+	struct llc_ctl *srv_ctl = &lgr->llc_ctl[LLC_SERV];
+	int initiator;
+
+	switch (qentry->msg.hd.type) {
+	case LLC_ADD_LINK:
+		if (list_empty(&lgr->list)) /* link group to be freed */
+			break;
+		if (lgr->role == SMC_CLNT) {
+			if (srv_ctl->active == LLC_GRP_ADD_LINK) {
+				/* some process is waiting for this response */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_add_link_work);
+			}
+		} else if (start_llc_agent(SMC_SERV, qentry)) {
+			/* else on SERV handle client suggest */
+			schedule_work(&lgr->llc_add_link_work);
+		}
+		break;
+	case LLC_DEL_LINK:
+		if (lgr->role == SMC_CLNT) {
+			if (srv_ctl->active) {
+				/* DEL LINK REQ during ADD LINK SEQ */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_del_link_work);
+			}
+		} else {
+			if (qentry->msg.hd.flags & LLC_FLAG_DEL_ALL) {
+				smc_terminate_conn(lgr);
+				kfree(qentry);
+			} else if ((srv_ctl->active == LLC_GRP_ADD_LINK) &&
+				   (!srv_ctl->qentry)) {
+				/* DEL LINK REQ during ADD LINK SEQ */
+				srv_ctl->qentry = qentry;
+				wake_up_interruptible(&lgr->llc_waiter);
+			} else if (start_llc_agent(SMC_SERV, qentry)) {
+				schedule_work(&lgr->llc_del_link_work);
+			}
+		}
+		break;
+	case LLC_CONFIRM_RKEY:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		if (start_llc_agent(initiator, qentry))
+			schedule_work(&lgr->llc_conf_rkey_work);
+		break;
+	case LLC_DELETE_RKEY:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		if (start_llc_agent(initiator, qentry))
+			schedule_work(&lgr->llc_del_rkey_work);
+		break;
+	case LLC_CONFIRM_RKEY_CONT:
+		initiator = (lgr->role == SMC_SERV) ? SMC_CLNT : SMC_SERV;
+		lgr->llc_ctl[initiator].qentry = qentry;
+		wake_up_interruptible(&lgr->llc_waiter);
+		break;
+	case LLC_CONFIRM_LINK:
+	case LLC_ADD_LINK_CONT:
+		if (lgr->llc_ctl[LLC_SERV].active) {
+			/* some process is waiting for this response */
+			srv_ctl->qentry = qentry;
+			wake_up_interruptible(&lgr->llc_waiter);
+		}
+		break;
+	case LLC_TEST_LINK:
+		qentry->msg.hd.flags |= LLC_FLAG_RESP;
+		llc_send_msg(qentry->link, &qentry->msg, LLC_NO_NOTIFY);
+		kfree(qentry);
+		break;
+	case LLC_NWM_DATA:
+		lgr->nwm_flags = qentry->msg.hd.flags;
+		if (qentry->msg.hd.flags & LLC_FLAG_HOSTNAME)
+			strncpy(lgr->nwm_data, qentry->msg.data,
+				sizeof(lgr->nwm_data));
+		else
+			memcpy(lgr->nwm_data, qentry->msg.data,
+			       sizeof(lgr->nwm_data));
+		kfree(qentry);
+		break;
+	default:
+		if (!(qentry->msg.hd.type & LLC_OPT_MSG_CTRL)) {
+			/* LLC protocol violation */
+			llc_send_del_link(lnk, NULL, 0,
+					  LLC_DEL_PROT_VIOL, LLC_NO_NOTIFY);
+			smc_terminate_conn(lgr);
+		}
+		kfree(qentry);
+		break;
+	}
+}
+
+void llc_event_worker(struct work_struct *work)
+{
+	struct smc_link_group *lgr = container_of(work, struct smc_link_group,
+						  llc_event_work);
+	struct llc_qentry *qentry;
+
+	if ((!lgr->llc_ctl[LLC_SERV].active) &&
+	    (lgr->delayed_q) &&
+		(atomic_read(&lgr->delayed_q->link->state) != SMC_LINK_FREED))
+		llc_handle_event(lgr->delayed_q, lgr);
+
+again:
+	spin_lock_bh(&lgr->llc_event_q_lock);
+	if (!list_empty(&lgr->llc_event_q)) {
+		qentry = list_first_entry(&lgr->llc_event_q, struct llc_qentry,
+					  list);
+		list_del_init(&qentry->list);
+		spin_unlock_bh(&lgr->llc_event_q_lock);
+		llc_handle_event(qentry, lgr);
+		goto again;
+	}
+	spin_unlock_bh(&lgr->llc_event_q_lock);
+}
+
diff --git a/net/smc/smc_llc.h b/net/smc/smc_llc.h
new file mode 100644
index 0000000..f47c499
--- /dev/null
+++ b/net/smc/smc_llc.h
@@ -0,0 +1,192 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  Definitions for LLC message handling
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Klaus Wacker <Klaus.Wacker@de.ibm.com>
+ *              Ursula Braun <Ursula Braun@de.ibm.com>
+ */
+
+#define LLC_DATA_LEN    40
+#define LLC_SEND_POSTED		0xffffffff
+
+#define LLC_FLAG_RESP		0x80
+#define LLC_FLAG_HOSTNAME	0x80
+#define LLC_FLAG_RKEY_DEL	0x40
+#define LLC_FLAG_RKEY_NEG	0x20
+#define LLC_FLAG_RKEY_RETRY	0x10
+#define LLC_FLAG_DEL_ORDERLY	0x20
+#define LLC_FLAG_DEL_ALL	0x40
+#define LLC_FLAG_ADD_LNK_REJ	0x40
+#define LLC_FLAG_NO_ALT_PATH	0x01
+#define LLC_FLAG_LOCAL		0x04    /* Linux-only enhancement */
+#define LLC_FLAGS_DEL_ALL_ORD	0x60
+
+/* LLC DELETE LINK Request Reason Codes */
+#define LLC_DEL_LOST_PATH	0x00010000
+#define LLC_DEL_OP_INIT_TERM	0x00020000
+#define LLC_DEL_PROG_INIT_TERM	0x00030000
+#define LLC_DEL_PROT_VIOL	0x00040000
+#define LLC_DEL_NO_ASYM_NEEDED	0x00050000
+
+enum llc_reqresp {
+	LLC_REQ,
+	LLC_RESP
+};
+
+enum llc_msg_type {
+	LLC_CONFIRM_LINK		= 0x01,
+	LLC_ADD_LINK			= 0x02,
+	LLC_ADD_LINK_CONT		= 0x03,
+	LLC_DEL_LINK			= 0x04,
+	LLC_CONFIRM_RKEY		= 0x06,
+	LLC_CONFIRM_RKEY_CONT		= 0x08,
+	LLC_DELETE_RKEY			= 0x09,
+	LLC_TEST_LINK			= 0x07,
+	LLC_OPT_MSG_CTRL		= 0x80,
+	LLC_NWM_DATA			= 0x8A,
+	RMBE_CTRL			= 0xFE,
+};
+
+struct rmb_rtoken {		/* RMB credentials given to peer */
+	__be32 rmb_key;		/* rkey */
+	__be64 rmb_vaddr;	/* RDMA virtual address */
+} __packed;
+
+struct rmb_rtoken2 {
+	__be32 rmb_key;
+	__be32 rmb_key_new;
+	__be64 rmb_vaddr_new;
+} __packed;
+
+struct rmb_rtoken3 {
+	u8   link_id;
+	__be32 rmb_key;
+	__be64 rmb_vaddr;
+} __packed;
+
+struct llc_hdr {
+	u8 type;
+	u8 length;	/* 44 */
+	u8 version;	/* 0x01 */
+	u8 flags;
+} __packed;
+
+struct llc_msg {
+	struct llc_hdr hd;
+	u8 data[LLC_DATA_LEN];
+} __packed;
+
+struct llc_confirm_msg {		/* type 0x01 */
+	struct llc_hdr hd;
+	u8 sender_mac[6];
+	u8 sender_gid[16];
+	u8 sender_qp_num[3];
+	u8 link_num;
+	__be32 link_uid;
+	u8 max_links;
+	u8 reserved[9];
+} __packed;
+
+struct llc_add_link_msg {		/* type 0x02 */
+	struct llc_hdr hd;
+	u8 sender_mac[6];
+	u8 reserved2[2];
+	u8 sender_gid[16];
+	u8 sender_qp_num[3];
+	u8 link_num;
+	u8 flags2;	/* QP mtu */
+	u8 initial_psn[3];
+	u8 reserved[8];
+} __packed;
+
+struct llc_add_link_cont_msg {	/* type 0x03 */
+	struct llc_hdr hd;
+	u8 link_num;
+	u8 num_rkeys;
+	u8 reserved2[2];
+	struct rmb_rtoken2 rt[2];
+	u8 reserved[4];
+} __packed;
+
+struct llc_del_link_msg {		/* type 0x04 */
+	struct llc_hdr hd;
+	u8 link_num;
+	__be32 reason;
+	u8 reserved[35];
+} __packed;
+
+struct llc_confirm_rkey_msg {	/* type 0x05 */
+	struct llc_hdr hd;
+	struct rmb_rtoken3 rt[3];
+	u8 reserved[1];
+} __packed;
+
+struct llc_conf_rkey_cont_msg {	/* type 0x08 */
+	struct llc_hdr hd;
+	u8 num_rkeys;
+	struct rmb_rtoken3 rt[3];
+} __packed;
+
+struct llc_delete_rkey_msg {	/* type 0x09 */
+	struct llc_hdr hd;
+	u8 num_rkeys;
+	u8 err_mask;
+	u8 reserved[2];
+	__be32 rk[8];
+	u8 reserved2[4];
+} __packed;
+
+struct llc_test_link_msg {		/* type 0x07 */
+	struct llc_hdr hd;
+	u8 user_data[16];
+	u8 reserved[24];
+} __packed;
+
+struct llc_nwm_data_msg {		/* type 0x8A */
+	struct llc_hdr hd;
+	u8 ident[16];
+	u8 reserved[24];
+} __packed;
+
+struct llc_add_link_fake_msg {		/* type 0x82 */
+	struct llc_hdr hd;
+	struct smc_ib_device *ibdev;
+	u8 port_num;
+	u8 reserved[31];
+} __packed;
+
+struct llc_qentry {
+	struct list_head list;
+	struct smc_link *link;
+	struct llc_msg msg;
+};
+
+void llc_init_workers(struct smc_link_group *);
+int llc_enqueue(struct smc_link *, struct llc_msg *);
+int llc_initiate(struct smc_link_group *, enum llc_msg_group);
+void llc_stop(struct smc_link_group *, int);
+struct llc_qentry *llc_wait(struct smc_link_group *, enum smc_role, int, u8);
+int llc_send_confirm_link(struct smc_link *, u8 *, union ib_gid *,
+			  enum llc_reqresp, enum llc_ctl_ind);
+int llc_do_confirm_rkey(struct smc_link *, struct rmb_rx_addrs *);
+int llc_send_test_link(struct smc_link *, u8 *);
+int llc_send_add_link(struct smc_link *, struct smc_roce_defs *,
+		      struct smc_link *, enum llc_reqresp, enum llc_ctl_ind);
+int llc_send_del_link(struct smc_link *, struct smc_link *, u8, u32,
+		      enum llc_ctl_ind);
+int llc_do_del_rkey(struct smc_link *, struct rmb_rx_addrs *);
+void llc_event_worker(struct work_struct *);
+void llc_process_add_link(struct work_struct *);
+void llc_process_del_link(struct work_struct *);
+void llc_process_confirm_rkey(struct work_struct *);
+void llc_process_delete_rkey(struct work_struct *);
+int llc_get_fail_cause(struct smc_link_group *, enum llc_ctl_ind,
+		       struct smc_sock *);
+int llc_cli_add_link(struct smc_link *, struct llc_qentry *, struct smc_sock *);
+int llc_srv_add_link(struct smc_link *, struct smc_roce_defs *,
+		     struct smc_sock *);
+void llc_kill_waiters(struct smc_link_group *);
+void llc_cancel_wait(struct smc_link *);
diff --git a/net/smc/smc_proc.c b/net/smc/smc_proc.c
new file mode 100644
index 0000000..d4bf6d6
--- /dev/null
+++ b/net/smc/smc_proc.c
@@ -0,0 +1,884 @@
+/*
+ *  Shared Memory Communications over RDMA (SMC-R) and ROCE
+ *
+ *  auxiliary functions
+ *
+ *  Copyright IBM Corp. 2015
+ *
+ *  Author(s):  Ursula Braun <ursula.braun@de.ibm.com>
+ *              based on prototype from Frank Blaschka
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/socket.h>
+#include <linux/pci.h>
+#include <linux/seq_file.h>
+#include <linux/debugfs.h>
+#include <linux/uaccess.h>
+#include <linux/workqueue.h>
+#include <linux/proc_fs.h>
+#include <linux/netdevice.h>
+#include <linux/if_vlan.h>
+#include <linux/inetdevice.h>
+#include <linux/types.h>
+#include <net/sock.h>
+#include <net/inet_sock.h>
+#include <net/tcp.h>
+#include <net/route.h>
+#include <asm/byteorder.h>
+#include <rdma/ib_verbs.h>
+#include <rdma/ib_addr.h>
+
+#include "af_smc.h"
+#include "smc_llc.h"
+
+struct smc_pnet_entry {
+	struct list_head list;
+	char dev_name[IFNAMSIZ];
+	char net_name[SMC_MAX_PNET_ID_LEN];
+	u8 port;
+	enum {
+		PNET_ETH,
+		PNET_IB
+	} type;
+};
+
+static struct proc_dir_entry *procfs_root;
+static struct proc_dir_entry *procfs_pnet_conf;
+static struct proc_dir_entry *procfs_buff_cnt;
+static struct proc_dir_entry *procfs_max_conn_per_lgr;
+static struct proc_dir_entry *procfs_port_add_delay;
+static struct proc_dir_entry *procfs_sndbuf;
+static struct proc_dir_entry *procfs_rcvbuf;
+static struct list_head smc_pnet_list;
+static rwlock_t smc_pnet_list_lock;
+
+/* determine first gid of roce device  and derive mac from it */
+int smc_get_ib_mac(struct smc_roce_defs *rocdefs)
+{
+	int rc = 0;
+
+	rc = ib_query_gid(rocdefs->ibdev->dev, rocdefs->port, 0, &rocdefs->gid);
+	if (!rc) {
+		memcpy(&rocdefs->mac[0], &rocdefs->gid.raw[8], 3);
+		memcpy(&rocdefs->mac[3], &rocdefs->gid.raw[13], 3);
+		rocdefs->mac[0] &= ~0x02;
+	}
+	return rc;
+}
+
+int smc_port_active(struct smc_ib_device *dev, u8 port)
+{
+	struct ib_port_attr props;
+	int rc;
+
+	rc = ib_query_port(dev->dev, port, &props);
+	if (rc)
+		return 0;
+	if (props.state != IB_PORT_ACTIVE)
+		return 0;
+	return 1;
+}
+
+static void smc_ib_dev_by_pnet(struct smc_roce_defs *rocdefs, char *pnet,
+			       struct smc_link *link)
+{
+	struct smc_ib_device *dev = NULL;
+	struct smc_pnet_entry *entry;
+	int found = 0;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if ((entry->type != PNET_IB) ||
+		    strncmp(pnet, entry->net_name, sizeof(entry->net_name)))
+			continue;
+		if (link &&
+		    !strncmp(entry->dev_name,
+			     link->roce_defs.ibdev->dev->name,
+			     sizeof(entry->dev_name)) &&
+		    (entry->port == link->roce_defs.port)) {
+			continue;
+		}
+		list_for_each_entry(dev, &smc_ib_devices, list) {
+			if (!strncmp(dev->dev->name, entry->dev_name,
+				     sizeof(entry->dev_name))) {
+				read_unlock(&smc_pnet_list_lock);
+				found = smc_port_active(dev, entry->port);
+				read_lock(&smc_pnet_list_lock);
+				if (found)
+					break;
+			}
+		}
+		if (found)
+			break;
+	}
+	read_unlock(&smc_pnet_list_lock);
+
+	if (!found)
+		return;
+
+	rocdefs->ibdev = dev;
+	rocdefs->port = entry->port;
+}
+
+int smc_pnet_by_ibdev(u8 port, char *pnet, struct smc_ib_device *dev)
+{
+	struct smc_pnet_entry *entry;
+	int rc = -EINVAL;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_IB &&
+		    !strncmp(dev->dev->name, entry->dev_name,
+			     sizeof(entry->dev_name)) &&
+		    (port == entry->port)) {
+			rc = 0;
+			strncpy(pnet, entry->net_name, sizeof(entry->net_name));
+			break;
+		}
+	}
+	read_unlock(&smc_pnet_list_lock);
+	return rc;
+}
+
+static int smc_pnet_by_tcpsk(char *pnet, struct sock *sk)
+{
+	int rc = -EINVAL;
+	struct smc_pnet_entry *entry;
+	struct dst_entry *dst = sk_dst_get(sk);
+
+	if (!dst)
+		goto out;
+	if (!dst->dev)
+		goto out_rel;
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_ETH &&
+		    !strncmp(dst->dev->name, entry->dev_name,
+			     sizeof(entry->dev_name))) {
+			strncpy(pnet, entry->net_name, sizeof(entry->net_name));
+			rc = 0;
+			break;
+		}
+	}
+	read_unlock(&smc_pnet_list_lock);
+out_rel:
+	dst_release(dst);
+out:
+	return rc;
+}
+
+int smc_gid_by_dev(struct smc_roce_defs *rocdefs)
+{
+	int rc = 0;
+
+	rc = ib_query_gid(rocdefs->ibdev->dev, rocdefs->port, 0,
+			  &rocdefs->gid);
+	rocdefs->sgid_idx = 0;
+	smc_get_ib_mac(rocdefs);
+	return rc;
+}
+
+static int smc_gid_by_tcpsk(struct smc_roce_defs *rocdefs, struct sock *sk)
+{
+	struct dst_entry *dst = sk_dst_get(sk);
+	int rc;
+
+	rocdefs->vlan = 0;
+	if (dst) {
+		if (dst->dev) {
+			if (is_vlan_dev(dst->dev))
+				rocdefs->vlan = vlan_dev_vlan_id(dst->dev);
+		}
+		dst_release(dst);
+	}
+	rc = smc_gid_by_dev(rocdefs);
+	return rc;
+}
+
+void smc_find_roce_resources(struct smc_roce_defs *rocdefs, char *pnet,
+			     struct sockaddr_in *inaddr, struct sock *sk)
+{
+	int rc;
+
+	rocdefs->ibdev = NULL;
+	rocdefs->port = 0;
+	rc = smc_pnet_by_tcpsk(pnet, sk);
+	if (rc) {
+		/* add fallback path and try to find ib dev/port
+		 * via mac address of tcp sock
+		 */
+		return;
+	}
+	smc_ib_dev_by_pnet(rocdefs, pnet, NULL);
+	if (!rocdefs->ibdev)
+		return;
+
+	rc = smc_gid_by_tcpsk(rocdefs, sk);
+	if (rc) {
+		rocdefs->ibdev = NULL;
+		return;
+	}
+}
+
+void smc_find_alt_roce_resources(struct smc_roce_defs *rocdefs,
+				 struct smc_link_group *lgr,
+				 struct smc_link *link)
+{
+	int rc;
+
+	smc_ib_dev_by_pnet(rocdefs, lgr->pnet_id, link);
+	if (!rocdefs->ibdev)
+		return;
+
+	rocdefs->vlan = lgr->vlan;
+	rc = smc_gid_by_dev(rocdefs);
+	if (rc) {
+			rocdefs->ibdev = NULL;
+			return;
+	}
+}
+
+int smc_netinfo_by_tcpsk(__be32 *subnet, u8 *mlen, struct socket *tcpsocket)
+{
+	struct dst_entry *dst = sk_dst_get(tcpsocket->sk);
+	struct sockaddr_in addr;
+	int rc = -ENODEV;
+	int len;
+
+	if (!dst)
+		goto out;
+	if (!dst->dev)
+		goto out_rel;
+		kernel_getsockname(tcpsocket,
+				   (struct sockaddr *)&addr, &len);
+	for_ifa(dst->dev->ip_ptr) {
+		if (ifa->ifa_address != addr.sin_addr.s_addr)
+			continue;
+		*mlen = inet_mask_len(ifa->ifa_mask);
+		*subnet = ifa->ifa_address & ifa->ifa_mask;
+		rc = 0;
+		break;
+	} endfor_ifa(dst->dev->ip_ptr);
+out_rel:
+	dst_release(dst);
+out:
+	return rc;
+}
+
+static void smc_print_area(struct seq_file *m, char *buf, int max_len,
+			   int pr_len)
+{
+	char fmtbuf[60];
+	int i, len;
+
+	len = min_t(size_t, max_len, pr_len);
+	while (len > 0) {
+		i = (len < 16) ? len : 16;
+		hex_dump_to_buffer(buf, i, 16, 4, fmtbuf, 60, 0);
+		seq_printf(m, "%p: %s\n", buf, fmtbuf);
+		buf += 16;
+		len -= 16;
+	}
+}
+
+static int smc_sock_show(struct seq_file *m, void *v)
+{
+	struct smc_proc_dir_entry *proc = m->private;
+	struct sock *sk;
+	struct smc_sock *smc;
+	struct sockaddr_in locl_addr, peer_addr;
+	int rc, len;
+
+	if (!proc)
+		return 0;
+
+	sk = proc->sk;
+	if (!sk)
+		return 0;
+
+	sock_hold(sk);
+	if (sock_flag(sk, SOCK_DEAD) || (sk->sk_state == SMC_DESTRUCT))
+		goto out;
+
+	smc = smc_sk(sk);
+	seq_printf(m, "state: %d\n", sk->sk_state);
+	seq_printf(m, "uid: %d\n",
+		   from_kuid_munged(seq_user_ns(m), sock_i_uid(sk)));
+	seq_printf(m, "inode: %ld\n", sock_i_ino(sk));
+
+	if (sk->sk_state == SMC_INIT)
+		goto out;
+
+	if ((smc->tcpsocket) && (smc->tcpsocket->sk)) {
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket,
+						  (struct sockaddr *)&locl_addr,
+						  &len, 0);
+		if (!rc) {
+			seq_printf(m, "local_addr: %pI4\n",
+				   &locl_addr.sin_addr.s_addr);
+			seq_printf(m, "local_port: %d\n",
+				   ntohs(locl_addr.sin_port));
+		}
+	}
+
+	if (sk->sk_state == SMC_LISTEN)
+		goto out;
+
+	if ((smc->tcpsocket) && (smc->tcpsocket->sk)) {
+		rc = smc->tcpsocket->ops->getname(smc->tcpsocket,
+						  (struct sockaddr *)&peer_addr,
+						  &len, 1);
+		if (!rc) {
+			seq_printf(m, "peer_addr: %pI4\n",
+				   &peer_addr.sin_addr.s_addr);
+			seq_printf(m, "peer_port: %d\n",
+				   ntohs(peer_addr.sin_port));
+		}
+	}
+
+	seq_printf(m, "tcp_fallback: %d\n", smc->use_tcp);
+	if (smc->use_tcp)
+		goto out;
+
+	if (smc->conn.lgr && (sk->sk_state != SMC_CLOSED)) {
+		seq_printf(m, "link group target: %#x\n",
+			   smc->conn.lgr->daddr);
+		seq_printf(m, "link group role: %d\n", smc->conn.lgr->role);
+		seq_puts(m, "gid_peer:\n");
+		smc_print_area(m, smc->conn.lgr->lnk[0].gid_peer, 16, 16);
+		smc_print_area(m, smc->conn.lgr->lnk[1].gid_peer, 16, 16);
+	}
+	seq_printf(m, "tx buffer len: %d\n", smc->conn.rmb_tx_size);
+	seq_printf(m, "rx buffer len: %d\n", smc->conn.rmb_rx_size);
+	seq_printf(m, "local alert token: %#x\n", smc->conn.alert_token_local);
+
+	seq_printf(m, "peer rx buffer len: %d\n", smc->conn.peer_rx_buf_len);
+	seq_printf(m, "local rx pc %d pw %d cc %d cw %d\n",
+		   smc->conn.local_rx_ctrl.p_curs.s.curs.c,
+		   smc->conn.local_rx_ctrl.p_curs.s.curs.w,
+		   smc->conn.local_rx_ctrl.c_curs.s.curs.c,
+		   smc->conn.local_rx_ctrl.c_curs.s.curs.w);
+	seq_printf(m, "local tx pc %d pw %d cc %d cw %d\n",
+		   smc->conn.local_tx_ctrl.p_curs.s.curs.c,
+		   smc->conn.local_tx_ctrl.p_curs.s.curs.w,
+		   smc->conn.local_tx_ctrl.c_curs.s.curs.c,
+		   smc->conn.local_tx_ctrl.c_curs.s.curs.w);
+	seq_printf(m, "tx producer_flags %#x conn_state_flags %#x\n",
+		   *(u8 *)&smc->conn.local_tx_ctrl.p_flags,
+		   *(u8 *)&smc->conn.local_tx_ctrl.conn_state_flags);
+	seq_printf(m, "rx producer_flags %#x conn_state_flags %#x\n",
+		   *(u8 *)&smc->conn.local_rx_ctrl.p_flags,
+		   *(u8 *)&smc->conn.local_rx_ctrl.conn_state_flags);
+
+	if (smc->conn.rmb_tx_elem &&
+	    smc->conn.rmb_tx_elem->rmb_tx &&
+	    smc->conn.rmb_tx_elem->rmb_tx) {
+		seq_puts(m, "tx buffer:\n");
+		smc_print_area(m, smc->conn.rmb_tx_elem->rmb_tx/*->buffer*/,
+			       smc->conn.rmb_tx_size, 15000);
+	}
+	if (smc->conn.rmb_rx_elem &&
+	    smc->conn.rmb_rx_elem->rmb_rx &&
+	    smc->conn.rmb_rx_elem->rmb_rx) {
+		seq_puts(m, "rx buffer:\n");
+		smc_print_area(m, smc->conn.rmb_rx_elem->rmb_rx/*->buffer*/,
+			       smc->conn.rmb_rx_size, 15000);
+	}
+
+out:
+	sock_put(sk);
+	return 0;
+}
+
+static int smc_sock_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_sock_show, PDE_DATA(inode));
+}
+
+static const struct file_operations procfs_smc_sock_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_sock_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.release = single_release,
+};
+
+int smc_sock_proc_create(struct sock *sk)
+{
+	struct smc_sock *smc = smc_sk(sk);
+
+	sprintf(smc->proc_name, "Sx%p_0x%p", sk, smc->proc);
+	smc->proc->proc = proc_create_data(smc->proc_name,
+					   S_IFREG | S_IRUGO | S_IWUSR,
+					   procfs_root, &procfs_smc_sock_fops,
+					   smc->proc);
+	return 0;
+}
+
+static int smc_pnet_show(struct seq_file *m, void *v)
+{
+	struct smc_pnet_entry *entry;
+
+	read_lock(&smc_pnet_list_lock);
+	list_for_each_entry(entry, &smc_pnet_list, list) {
+		if (entry->type == PNET_IB)
+			seq_printf(m, "%-*s ib  %s %d\n", SMC_MAX_PNET_ID_LEN,
+				   entry->net_name, entry->dev_name,
+				   entry->port);
+		else
+			seq_printf(m, "%-*s eth %s\n", SMC_MAX_PNET_ID_LEN,
+				   entry->net_name, entry->dev_name);
+	}
+	read_unlock(&smc_pnet_list_lock);
+	return 0;
+}
+
+static struct smc_pnet_entry *smc_pnet_create_entry(int argc, char *argv[])
+{
+	struct smc_pnet_entry *entry;
+	long rc = 0;
+
+	entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+	if (!entry) {
+		rc = -ENOMEM;
+		goto out;
+	}
+	rc = -EINVAL;
+	if (!strcmp(argv[2], "ib")) {
+		if ((argc != 5) || kstrtou8(argv[4], 0, &entry->port))
+			goto out;
+		entry->type = PNET_IB;
+	} else if (!strcmp(argv[2], "eth")) {
+		if (argc > 4)
+			goto out;
+		entry->type = PNET_ETH;
+	} else {
+		goto out;
+	}
+	strncpy(entry->net_name, argv[1], sizeof(entry->net_name));
+	strncpy(entry->dev_name, argv[3], sizeof(entry->dev_name));
+
+	return entry;
+
+out:
+	kfree(entry);
+	return ERR_PTR(rc);
+}
+
+/* Update the PNET ID table. Make sure entries are sorted by PNET ID
+ * for fancy output.
+ * @entry: entry to search for
+ * @del: delete existing entry if found, otherwise add.
+ *
+ * Returns 0 on success, <0 otherwise.
+ */
+static int smc_pnet_update(struct smc_pnet_entry *entry, bool del)
+{
+	struct smc_pnet_entry *tmp_entry, *n_entry;
+	int rc = -EINVAL;
+	int found = 0;
+
+	/* keep list sorted by PNET ID */
+	write_lock(&smc_pnet_list_lock);
+	list_for_each_entry_safe(tmp_entry, n_entry, &smc_pnet_list, list) {
+		if (entry->type == tmp_entry->type &&
+		    !strncmp(entry->dev_name, tmp_entry->dev_name,
+		    sizeof(entry->dev_name)) &&
+		    entry->port == tmp_entry->port) {
+			/* entry for this device, port and type found */
+			if (del) {
+				list_del(&tmp_entry->list);
+				kfree(tmp_entry);
+				kfree(entry);
+				rc = 0;
+			}
+			found = 1;
+			break;
+		}
+		if (strncmp(entry->net_name, tmp_entry->net_name,
+			    sizeof(entry->net_name)) < 0)
+			break;
+	}
+	if (!del && !found) {
+		list_add_tail(&entry->list, &tmp_entry->list);
+		rc = 0;
+	}
+	write_unlock(&smc_pnet_list_lock);
+
+	return rc;
+}
+
+/* add|del <PNET ID> eth|ib <device name> [<port>] */
+static ssize_t smc_pnet_write(struct file *file,
+			      const char __user *user_buffer,
+			      size_t count, loff_t *offset)
+{
+	char buf[128];
+	char *pos;
+	char *tok;
+	char *tmp;
+	char *argv[5];
+	int argc, rc = -EINVAL, del;
+	size_t len = min_t(size_t, count, sizeof(buf) - 1);
+	struct smc_pnet_entry *entry;
+
+	memset(buf, 0, sizeof(buf));
+	argc = 0;
+
+	if (copy_from_user(buf, user_buffer, len)) {
+		rc = -EFAULT;
+		goto out;
+	}
+	tmp = buf;
+	pos = strsep(&tmp, "\n");
+
+	while ((tok = strsep(&pos, " "))) {
+		if (!strcmp(tok, ""))
+			continue;
+		if (argc == 5)
+			break;
+		argv[argc] = tok;
+		argc++;
+	}
+
+	if (argc < 4 || tok)
+		goto out;
+
+	if (!strcmp(argv[0], "add"))
+		del = 0;
+	else if (!strcmp(argv[0], "del"))
+		del = 1;
+	else
+		goto out;
+	entry = smc_pnet_create_entry(argc, argv);
+	if (IS_ERR(entry)) {
+		rc = PTR_ERR(entry);
+		goto out;
+	}
+	rc = smc_pnet_update(entry, del);
+	if (!rc)
+		rc = count;
+out:
+	return rc;
+}
+
+static int smc_pnet_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_pnet_show, PDE_DATA(inode));
+}
+
+static int smc_buff_cnt_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%u\n", smc_ctrl_buf_cnt);
+	return 0;
+}
+
+static ssize_t smc_buff_cnt_write(struct file *file,
+				  const char __user *user_buffer,
+				  size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 3) ||
+	    (tmp > SMC_MAX_WRE) ||
+	    ((smc_max_conn_per_lgr * 10 * tmp) > SMC_MAX_CQE)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	spin_lock_bh(&smc_lgr_list.lock);
+	if (!list_empty(&smc_lgr_list.list)) {
+		rc = -EPERM;
+		goto out_unlock;
+	}
+	smc_ctrl_buf_cnt = tmp;
+	rc = count;
+out_unlock:
+	spin_unlock_bh(&smc_lgr_list.lock);
+out:
+	return rc;
+}
+
+static int smc_buff_cnt_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_buff_cnt_show, PDE_DATA(inode));
+}
+
+static int smc_max_conn_per_lgr_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_max_conn_per_lgr);
+	return 0;
+}
+
+static ssize_t smc_max_conn_per_lgr_write(struct file *file,
+					  const char __user *user_buffer,
+					  size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 1) ||
+	    (tmp > SMC_MAX_RMB) ||
+	    ((tmp * 10 * smc_ctrl_buf_cnt) > SMC_MAX_CQE)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_max_conn_per_lgr = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_max_conn_per_lgr_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_max_conn_per_lgr_show, PDE_DATA(inode));
+}
+
+static int smc_port_add_delay_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_port_add_delay);
+	return 0;
+}
+
+static ssize_t smc_port_add_delay_write(struct file *file,
+					const char __user *user_buffer,
+					size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < 0) ||
+	    (tmp > SMC_MAX_PORT_ADD_DELAY)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_port_add_delay = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_port_add_delay_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_port_add_delay_show, PDE_DATA(inode));
+}
+
+static int smc_sndbuf_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_def_sndbuf_size);
+	return 0;
+}
+
+static ssize_t smc_sndbuf_write(struct file *file,
+				const char __user *user_buffer, size_t count,
+				loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < SK_MEM_QUANTUM) ||
+	    (tmp > sysctl_wmem_max)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_def_sndbuf_size = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_sndbuf_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_sndbuf_show, PDE_DATA(inode));
+}
+
+static int smc_rcvbuf_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", smc_def_rcvbuf_size);
+	return 0;
+}
+
+static ssize_t smc_rcvbuf_write(struct file *file,
+				const char __user *user_buffer,
+				size_t count, loff_t *offset)
+{
+	unsigned int tmp;
+	int cnt, rc = 0;
+
+	cnt = kstrtouint_from_user(user_buffer, count, 0,  &tmp);
+	if ((cnt < 0) ||
+	    (tmp < SK_MEM_QUANTUM) ||
+	    (tmp > sysctl_rmem_max)) {
+		rc = -EINVAL;
+		goto out;
+	}
+	smc_def_rcvbuf_size = tmp;
+	rc = count;
+out:
+	return rc;
+}
+
+static int smc_rcvbuf_seq_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, smc_rcvbuf_show, PDE_DATA(inode));
+}
+
+static const struct file_operations procfs_pnet_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_pnet_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_pnet_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_buff_cnt_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_buff_cnt_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_buff_cnt_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_max_conn_per_lgr_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_max_conn_per_lgr_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_max_conn_per_lgr_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_port_add_delay_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_port_add_delay_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_port_add_delay_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_sndbuf_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_sndbuf_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_sndbuf_write,
+	.release = single_release,
+};
+
+static const struct file_operations procfs_rcvbuf_fops = {
+	.owner	 = THIS_MODULE,
+	.open	 = smc_rcvbuf_seq_open,
+	.read	 = seq_read,
+	.llseek	 = seq_lseek,
+	.write   = smc_rcvbuf_write,
+	.release = single_release,
+};
+
+int smc_proc_init(void)
+{
+	INIT_LIST_HEAD(&smc_pnet_list);
+	rwlock_init(&smc_pnet_list_lock);
+	procfs_root = proc_net_mkdir(&init_net, "smc", init_net.proc_net);
+	if (!procfs_root)
+		goto out;
+	procfs_pnet_conf =
+		proc_create_data("pnet_conf", S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_pnet_fops, NULL);
+	if (!procfs_pnet_conf)
+		goto out_smc;
+	procfs_buff_cnt =
+		proc_create_data("ctrl_buffer_count",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_buff_cnt_fops, NULL);
+	if (!procfs_buff_cnt)
+		goto out_pnet_conf;
+	procfs_max_conn_per_lgr =
+		proc_create_data("max_conn_per_lgr",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_max_conn_per_lgr_fops,
+				 NULL);
+	if (!procfs_max_conn_per_lgr)
+		goto out_ctrl_buffer_count;
+	procfs_port_add_delay =
+		proc_create_data("port_add_delay",
+				 S_IFREG | S_IRUGO | S_IWUSR,
+				 procfs_root, &procfs_port_add_delay_fops,
+				 NULL);
+	if (!procfs_port_add_delay)
+		goto out_max_conn_per_lgr;
+	procfs_sndbuf = proc_create_data("sndbuf", S_IFREG | S_IRUGO | S_IWUSR,
+					 procfs_root, &procfs_sndbuf_fops,
+					 NULL);
+	if (!procfs_sndbuf)
+		goto out_port_add_delay;
+	procfs_rcvbuf = proc_create_data("rcvbuf", S_IFREG | S_IRUGO | S_IWUSR,
+					 procfs_root, &procfs_rcvbuf_fops,
+					 NULL);
+	if (!procfs_rcvbuf)
+		goto out_sndbuf;
+	return 0;
+
+out_sndbuf:
+	remove_proc_entry("sndbuf", procfs_root);
+out_port_add_delay:
+	remove_proc_entry("port_add_delay", procfs_root);
+out_max_conn_per_lgr:
+	remove_proc_entry("max_conn_per_lgr", procfs_root);
+out_ctrl_buffer_count:
+	remove_proc_entry("ctrl_buffer_count", procfs_root);
+out_pnet_conf:
+	remove_proc_entry("pnet_conf", procfs_root);
+out_smc:
+	remove_proc_entry("smc", init_net.proc_net);
+out:
+	return -EFAULT;
+}
+
+void smc_proc_exit(void)
+{
+	struct smc_pnet_entry *entry, *tmp_entry;
+
+	write_lock(&smc_pnet_list_lock);
+	list_for_each_entry_safe(entry, tmp_entry, &smc_pnet_list, list) {
+		list_del(&entry->list);
+		kfree(entry);
+	}
+	write_unlock(&smc_pnet_list_lock);
+
+	if (procfs_rcvbuf)
+		remove_proc_entry("rcvbuf", procfs_root);
+	if (procfs_sndbuf)
+		remove_proc_entry("sndbuf", procfs_root);
+	if (procfs_port_add_delay)
+		remove_proc_entry("port_add_delay", procfs_root);
+	if (procfs_max_conn_per_lgr)
+		remove_proc_entry("max_conn_per_lgr", procfs_root);
+	if (procfs_buff_cnt)
+		remove_proc_entry("ctrl_buffer_count", procfs_root);
+	if (procfs_pnet_conf)
+		remove_proc_entry("pnet_conf", procfs_root);
+	if (procfs_root)
+		remove_proc_entry("smc", init_net.proc_net);
+}
-- 
2.3.8