Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[David Miller]

From: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Date: Wed, 11 Jun 2008 17:57:18 +0400

> Major issue is that tcp_defer_accept_check() manipulates with not locked
> listening socket. And from all that I know it is impossible to take
> the lock in this context.
> 
> Also I see no accounting for those sockets. With this patch any server, which
> set deferred accept, can be flooded with sockets until memory exhausts.
> I did not test and would be glad to be mistaken.
> 
> 
> Issue with locking can be solved by adding a separate spinlock for
> manipulations with accept_queue. Apparently, accounting and killing
> sockets, which become stale after closing listening socket and
> are going to be alive for up to 65535 seconds, also goes under this lock.
> 
> Frankly, cost looks too high for this feature.
> 
> Hiding from accept() sockets with only out-of-order data only
> is the only thing which is impossible with old approach. Is this really
> so valuable? My opinion: no, this is nothing but a new loophole
> to consume memory without control.

Yes, we discussed the locking issue over past few days.  See
the thread: "stuck localhost TCP connections, v2.6.26-rc3+"

More and more, the arguments are mounting to completely revert the
established code path changes, and frankly that is likely what I am
going to do by the end of today.

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Wed, 11 Jun 2008 16:52:55 -0700 (PDT)

> More and more, the arguments are mounting to completely revert the
> established code path changes, and frankly that is likely what I am
> going to do by the end of today.

Here is the revert patch I intend to send to Linus:

tcp: Revert 'process defer accept as established' changes.

This reverts two changesets, ec3c0982a2dd1e671bad8e9d26c28dcba0039d87
("[TCP]: TCP_DEFER_ACCEPT updates - process as established") and
the follow-on bug fix 9ae27e0adbf471c7a6b80102e38e1d5a346b3b38
("tcp: Fix slab corruption with ipv6 and tcp6fuzz").

This change causes several problems, first reported by Ingo Molnar
as a distcc-over-loopback regression where connections were getting
stuck.

Ilpo Järvinen first spotted the locking problems.  The new function
added by this code, tcp_defer_accept_check(), only has the
child socket locked, yet it is modifying state of the parent
listening socket.

Fixing that is non-trivial at best, because we can't simply just grab
the parent listening socket lock at this point, because it would
create an ABBA deadlock.  The normal ordering is parent listening
socket --> child socket, but this code path would require the
reverse lock ordering.

Next is a problem noticed by Vitaliy Gusev, he noted:

----------------------------------------
>--- a/net/ipv4/tcp_timer.c
>+++ b/net/ipv4/tcp_timer.c
>@@ -481,6 +481,11 @@ static void tcp_keepalive_timer (unsigned long data)
> 		goto death;
> 	}
>
>+	if (tp->defer_tcp_accept.request && sk->sk_state == TCP_ESTABLISHED) {
>+		tcp_send_active_reset(sk, GFP_ATOMIC);
>+		goto death;

Here socket sk is not attached to listening socket's request queue. tcp_done()
will not call inet_csk_destroy_sock() (and tcp_v4_destroy_sock() which should
release this sk) as socket is not DEAD. Therefore socket sk will be lost for
freeing.
----------------------------------------

Finally, Alexey Kuznetsov argues that there might not even be any
real value or advantage to these new semantics even if we fix all
of the bugs:

----------------------------------------
Hiding from accept() sockets with only out-of-order data only
is the only thing which is impossible with old approach. Is this really
so valuable? My opinion: no, this is nothing but a new loophole
to consume memory without control.
----------------------------------------

So revert this thing for now.

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 include/linux/tcp.h             |    7 ------
 include/net/request_sock.h      |    4 +-
 include/net/tcp.h               |    1 -
 net/ipv4/inet_connection_sock.c |   11 +++++++--
 net/ipv4/tcp.c                  |   18 +++++++++------
 net/ipv4/tcp_input.c            |   45 ---------------------------------------
 net/ipv4/tcp_ipv4.c             |    8 -------
 net/ipv4/tcp_minisocks.c        |   32 ++++++++++-----------------
 net/ipv4/tcp_timer.c            |    5 ----
 9 files changed, 33 insertions(+), 98 deletions(-)

diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 18e62e3..b31b6b7 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -239,11 +239,6 @@ static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
 	return (struct tcp_request_sock *)req;
 }
 
-struct tcp_deferred_accept_info {
-	struct sock *listen_sk;
-	struct request_sock *request;
-};
-
 struct tcp_sock {
 	/* inet_connection_sock has to be the first member of tcp_sock */
 	struct inet_connection_sock	inet_conn;
@@ -379,8 +374,6 @@ struct tcp_sock {
 	unsigned int		keepalive_intvl;  /* time interval between keep alive probes */
 	int			linger2;
 
-	struct tcp_deferred_accept_info defer_tcp_accept;
-
 	unsigned long last_synq_overflow; 
 
 	u32	tso_deferred;
diff --git a/include/net/request_sock.h b/include/net/request_sock.h
index b220b5f..0c96e7b 100644
--- a/include/net/request_sock.h
+++ b/include/net/request_sock.h
@@ -115,8 +115,8 @@ struct request_sock_queue {
 	struct request_sock	*rskq_accept_head;
 	struct request_sock	*rskq_accept_tail;
 	rwlock_t		syn_wait_lock;
-	u16			rskq_defer_accept;
-	/* 2 bytes hole, try to pack */
+	u8			rskq_defer_accept;
+	/* 3 bytes hole, try to pack */
 	struct listen_sock	*listen_opt;
 };
 
diff --git a/include/net/tcp.h b/include/net/tcp.h
index d448310..cf54034 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -139,7 +139,6 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define MAX_TCP_KEEPINTVL	32767
 #define MAX_TCP_KEEPCNT		127
 #define MAX_TCP_SYNCNT		127
-#define MAX_TCP_ACCEPT_DEFERRED 65535
 
 #define TCP_SYNQ_INTERVAL	(HZ/5)	/* Period of SYNACK timer */
 
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 828ea21..045e799 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -419,7 +419,8 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
 	struct inet_connection_sock *icsk = inet_csk(parent);
 	struct request_sock_queue *queue = &icsk->icsk_accept_queue;
 	struct listen_sock *lopt = queue->listen_opt;
-	int thresh = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+	int max_retries = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+	int thresh = max_retries;
 	unsigned long now = jiffies;
 	struct request_sock **reqp, *req;
 	int i, budget;
@@ -455,6 +456,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
 		}
 	}
 
+	if (queue->rskq_defer_accept)
+		max_retries = queue->rskq_defer_accept;
+
 	budget = 2 * (lopt->nr_table_entries / (timeout / interval));
 	i = lopt->clock_hand;
 
@@ -462,8 +466,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
 		reqp=&lopt->syn_table[i];
 		while ((req = *reqp) != NULL) {
 			if (time_after_eq(now, req->expires)) {
-				if (req->retrans < thresh &&
-				    !req->rsk_ops->rtx_syn_ack(parent, req)) {
+				if ((req->retrans < (inet_rsk(req)->acked ? max_retries : thresh)) &&
+				    (inet_rsk(req)->acked ||
+				     !req->rsk_ops->rtx_syn_ack(parent, req))) {
 					unsigned long timeo;
 
 					if (req->retrans++ == 0)
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index ab66683..fc54a48 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -2112,12 +2112,15 @@ static int do_tcp_setsockopt(struct sock *sk, int level,
 		break;
 
 	case TCP_DEFER_ACCEPT:
-		if (val < 0) {
-			err = -EINVAL;
-		} else {
-			if (val > MAX_TCP_ACCEPT_DEFERRED)
-				val = MAX_TCP_ACCEPT_DEFERRED;
-			icsk->icsk_accept_queue.rskq_defer_accept = val;
+		icsk->icsk_accept_queue.rskq_defer_accept = 0;
+		if (val > 0) {
+			/* Translate value in seconds to number of
+			 * retransmits */
+			while (icsk->icsk_accept_queue.rskq_defer_accept < 32 &&
+			       val > ((TCP_TIMEOUT_INIT / HZ) <<
+				       icsk->icsk_accept_queue.rskq_defer_accept))
+				icsk->icsk_accept_queue.rskq_defer_accept++;
+			icsk->icsk_accept_queue.rskq_defer_accept++;
 		}
 		break;
 
@@ -2299,7 +2302,8 @@ static int do_tcp_getsockopt(struct sock *sk, int level,
 			val = (val ? : sysctl_tcp_fin_timeout) / HZ;
 		break;
 	case TCP_DEFER_ACCEPT:
-		val = icsk->icsk_accept_queue.rskq_defer_accept;
+		val = !icsk->icsk_accept_queue.rskq_defer_accept ? 0 :
+			((TCP_TIMEOUT_INIT / HZ) << (icsk->icsk_accept_queue.rskq_defer_accept - 1));
 		break;
 	case TCP_WINDOW_CLAMP:
 		val = tp->window_clamp;
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index eba873e..cad73b7 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4541,49 +4541,6 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, struct tcphdr *th)
 	}
 }
 
-static int tcp_defer_accept_check(struct sock *sk)
-{
-	struct tcp_sock *tp = tcp_sk(sk);
-
-	if (tp->defer_tcp_accept.request) {
-		int queued_data =  tp->rcv_nxt - tp->copied_seq;
-		int hasfin =  !skb_queue_empty(&sk->sk_receive_queue) ?
-			tcp_hdr((struct sk_buff *)
-				sk->sk_receive_queue.prev)->fin : 0;
-
-		if (queued_data && hasfin)
-			queued_data--;
-
-		if (queued_data &&
-		    tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
-			if (sock_flag(sk, SOCK_KEEPOPEN)) {
-				inet_csk_reset_keepalive_timer(sk,
-							       keepalive_time_when(tp));
-			} else {
-				inet_csk_delete_keepalive_timer(sk);
-			}
-
-			inet_csk_reqsk_queue_add(
-				tp->defer_tcp_accept.listen_sk,
-				tp->defer_tcp_accept.request,
-				sk);
-
-			tp->defer_tcp_accept.listen_sk->sk_data_ready(
-				tp->defer_tcp_accept.listen_sk, 0);
-
-			sock_put(tp->defer_tcp_accept.listen_sk);
-			sock_put(sk);
-			tp->defer_tcp_accept.listen_sk = NULL;
-			tp->defer_tcp_accept.request = NULL;
-		} else if (hasfin ||
-			   tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
-			tcp_reset(sk);
-			return -1;
-		}
-	}
-	return 0;
-}
-
 static int tcp_copy_to_iovec(struct sock *sk, struct sk_buff *skb, int hlen)
 {
 	struct tcp_sock *tp = tcp_sk(sk);
@@ -4944,8 +4901,6 @@ step5:
 
 	tcp_data_snd_check(sk);
 	tcp_ack_snd_check(sk);
-
-	tcp_defer_accept_check(sk);
 	return 0;
 
 csum_error:
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 4f8485c..97a2300 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1918,14 +1918,6 @@ int tcp_v4_destroy_sock(struct sock *sk)
 		sk->sk_sndmsg_page = NULL;
 	}
 
-	if (tp->defer_tcp_accept.request) {
-		reqsk_free(tp->defer_tcp_accept.request);
-		sock_put(tp->defer_tcp_accept.listen_sk);
-		sock_put(sk);
-		tp->defer_tcp_accept.listen_sk = NULL;
-		tp->defer_tcp_accept.request = NULL;
-	}
-
 	atomic_dec(&tcp_sockets_allocated);
 
 	return 0;
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 019c8c1..8245247 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -571,8 +571,10 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
 	   does sequence test, SYN is truncated, and thus we consider
 	   it a bare ACK.
 
-	   Both ends (listening sockets) accept the new incoming
-	   connection and try to talk to each other. 8-)
+	   If icsk->icsk_accept_queue.rskq_defer_accept, we silently drop this
+	   bare ACK.  Otherwise, we create an established connection.  Both
+	   ends (listening sockets) accept the new incoming connection and try
+	   to talk to each other. 8-)
 
 	   Note: This case is both harmless, and rare.  Possibility is about the
 	   same as us discovering intelligent life on another plant tomorrow.
@@ -640,6 +642,13 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
 		if (!(flg & TCP_FLAG_ACK))
 			return NULL;
 
+		/* If TCP_DEFER_ACCEPT is set, drop bare ACK. */
+		if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
+		    TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
+			inet_rsk(req)->acked = 1;
+			return NULL;
+		}
+
 		/* OK, ACK is valid, create big socket and
 		 * feed this segment to it. It will repeat all
 		 * the tests. THIS SEGMENT MUST MOVE SOCKET TO
@@ -678,24 +687,7 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
 		inet_csk_reqsk_queue_unlink(sk, req, prev);
 		inet_csk_reqsk_queue_removed(sk, req);
 
-		if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
-		    TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
-
-			/* the accept queue handling is done is est recv slow
-			 * path so lets make sure to start there
-			 */
-			tcp_sk(child)->pred_flags = 0;
-			sock_hold(sk);
-			sock_hold(child);
-			tcp_sk(child)->defer_tcp_accept.listen_sk = sk;
-			tcp_sk(child)->defer_tcp_accept.request = req;
-
-			inet_csk_reset_keepalive_timer(child,
-						       inet_csk(sk)->icsk_accept_queue.rskq_defer_accept * HZ);
-		} else {
-			inet_csk_reqsk_queue_add(sk, req, child);
-		}
-
+		inet_csk_reqsk_queue_add(sk, req, child);
 		return child;
 
 	listen_overflow:
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 4de68cf..63ed9d6 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -489,11 +489,6 @@ static void tcp_keepalive_timer (unsigned long data)
 		goto death;
 	}
 
-	if (tp->defer_tcp_accept.request && sk->sk_state == TCP_ESTABLISHED) {
-		tcp_send_active_reset(sk, GFP_ATOMIC);
-		goto death;
-	}
-
 	if (!sock_flag(sk, SOCK_KEEPOPEN) || sk->sk_state == TCP_CLOSE)
 		goto out;
 
-- 
1.5.5.1.308.g1fbb5

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: David Miller <davem@davemloft.net>
> Date: Wed, 11 Jun 2008 16:52:55 -0700 (PDT)
> 
> > More and more, the arguments are mounting to completely revert the 
> > established code path changes, and frankly that is likely what I am 
> > going to do by the end of today.
> 
> Here is the revert patch I intend to send to Linus:
> 
> tcp: Revert 'process defer accept as established' changes.
> 
> This reverts two changesets, ec3c0982a2dd1e671bad8e9d26c28dcba0039d87
> ("[TCP]: TCP_DEFER_ACCEPT updates - process as established") and
> the follow-on bug fix 9ae27e0adbf471c7a6b80102e38e1d5a346b3b38
> ("tcp: Fix slab corruption with ipv6 and tcp6fuzz").
> 
> This change causes several problems, first reported by Ingo Molnar
> as a distcc-over-loopback regression where connections were getting
> stuck.
> 
> Ilpo Järvinen first spotted the locking problems.  The new function
> added by this code, tcp_defer_accept_check(), only has the
> child socket locked, yet it is modifying state of the parent
> listening socket.
> 
> Fixing that is non-trivial at best, because we can't simply just grab
> the parent listening socket lock at this point, because it would
> create an ABBA deadlock.  The normal ordering is parent listening
> socket --> child socket, but this code path would require the
> reverse lock ordering.
> 
> Next is a problem noticed by Vitaliy Gusev, he noted:
> 
> ----------------------------------------
> >--- a/net/ipv4/tcp_timer.c
> >+++ b/net/ipv4/tcp_timer.c
> >@@ -481,6 +481,11 @@ static void tcp_keepalive_timer (unsigned long data)
> > 		goto death;
> > 	}
> >
> >+	if (tp->defer_tcp_accept.request && sk->sk_state == TCP_ESTABLISHED) {
> >+		tcp_send_active_reset(sk, GFP_ATOMIC);
> >+		goto death;
> 
> Here socket sk is not attached to listening socket's request queue. tcp_done()
> will not call inet_csk_destroy_sock() (and tcp_v4_destroy_sock() which should
> release this sk) as socket is not DEAD. Therefore socket sk will be lost for
> freeing.
> ----------------------------------------
> 
> Finally, Alexey Kuznetsov argues that there might not even be any
> real value or advantage to these new semantics even if we fix all
> of the bugs:
> 
> ----------------------------------------
> Hiding from accept() sockets with only out-of-order data only
> is the only thing which is impossible with old approach. Is this really
> so valuable? My opinion: no, this is nothing but a new loophole
> to consume memory without control.
> ----------------------------------------
> 
> So revert this thing for now.
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>

the 3 reverts have been extensively tested in -tip via:

 # tip/out-of-tree: 9e5b6ca: tcp: revert DEFER_ACCEPT modifications

and the distcc problems are fixed. (The locking fix alone did not fix it 
conclusively in my testing, possibly due to the follow-on observations 
outlined in your description.)

Tested-by: Ingo Molnar <mingo@elte.hu>

	Ingo

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Fri, 13 Jun 2008 08:30:37 +0200

> the 3 reverts have been extensively tested in -tip via:
> 
>  # tip/out-of-tree: 9e5b6ca: tcp: revert DEFER_ACCEPT modifications
> 
> and the distcc problems are fixed. (The locking fix alone did not fix it 
> conclusively in my testing, possibly due to the follow-on observations 
> outlined in your description.)
> 
> Tested-by: Ingo Molnar <mingo@elte.hu>

I didn't revert all three changes, just the final part of that
3 part series.

Please test the patch I actually applied.

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Fri, 13 Jun 2008 08:30:37 +0200
> 
> > the 3 reverts have been extensively tested in -tip via:
> > 
> >  # tip/out-of-tree: 9e5b6ca: tcp: revert DEFER_ACCEPT modifications
> > 
> > and the distcc problems are fixed. (The locking fix alone did not fix it 
> > conclusively in my testing, possibly due to the follow-on observations 
> > outlined in your description.)
> > 
> > Tested-by: Ingo Molnar <mingo@elte.hu>
> 
> I didn't revert all three changes, just the final part of that 3 part 
> series.
> 
> Please test the patch I actually applied.

i just updated all my testsystems to revert the change i tested so far, 
and updated it to yours. The delta between the two is the 3 lines patch 
below.

A few testsystems already booted into your patch, so if i dont report a 
hung TCP connection in the next 6 hours consider it:

Tested-by: Ingo Molnar <mingo@elte.hu>

	Ingo

--------------->
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index ec83448..045e799 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -466,9 +466,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
 		reqp=&lopt->syn_table[i];
 		while ((req = *reqp) != NULL) {
 			if (time_after_eq(now, req->expires)) {
-				if ((req->retrans < thresh ||
-				     (inet_rsk(req)->acked && req->retrans < max_retries))
-				    && !req->rsk_ops->rtx_syn_ack(parent, req)) {
+				if ((req->retrans < (inet_rsk(req)->acked ? max_retries : thresh)) &&
+				    (inet_rsk(req)->acked ||
+				     !req->rsk_ops->rtx_syn_ack(parent, req))) {
 					unsigned long timeo;
 
 					if (req->retrans++ == 0)

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 2489 bytes --]


* Ingo Molnar <mingo@elte.hu> wrote:

> > Please test the patch I actually applied.
> 
> i just updated all my testsystems to revert the change i tested so 
> far, and updated it to yours. The delta between the two is the 3 lines 
> patch below.
> 
> A few testsystems already booted into your patch, so if i dont report 
> a hung TCP connection in the next 6 hours consider it:
> 
> Tested-by: Ingo Molnar <mingo@elte.hu>

this threw the warning below - never saw that before in thousands of 
bootups and this was the only networking change that happened. config 
and bootlog attached. Might be unlucky coincidence.

	Ingo

[  173.354049] NETDEV WATCHDOG: eth0: transmit timed out
[  173.354148] ------------[ cut here ]------------
[  173.354221] WARNING: at net/sched/sch_generic.c:222 dev_watchdog+0x9a/0xec()
[  173.354298] Modules linked in:
[  173.354421] Pid: 13452, comm: cc1 Tainted: G        W 2.6.26-rc6-00273-g81ae43a-dirty #2573
[  173.354516]  [<c01250ca>] warn_on_slowpath+0x46/0x76
[  173.354641]  [<c011d428>] ? try_to_wake_up+0x1d6/0x1e0
[  173.354815]  [<c01411e9>] ? trace_hardirqs_off+0xb/0xd
[  173.357370]  [<c011d43d>] ? default_wake_function+0xb/0xd
[  173.357370]  [<c014112a>] ? trace_hardirqs_off_caller+0x15/0xc9
[  173.357370]  [<c01411e9>] ? trace_hardirqs_off+0xb/0xd
[  173.357370]  [<c0142c83>] ? trace_hardirqs_on+0xb/0xd
[  173.357370]  [<c0142b33>] ? trace_hardirqs_on_caller+0x16/0x15b
[  173.357370]  [<c0142c83>] ? trace_hardirqs_on+0xb/0xd
[  173.357370]  [<c06bb3c9>] ? _spin_unlock_irqrestore+0x5b/0x71
[  173.357370]  [<c0133d46>] ? __queue_work+0x2d/0x32
[  173.357370]  [<c0134023>] ? queue_work+0x50/0x72
[  173.357483]  [<c0134059>] ? schedule_work+0x14/0x16
[  173.357654]  [<c05c59b8>] dev_watchdog+0x9a/0xec
[  173.357783]  [<c012d456>] run_timer_softirq+0x13d/0x19d
[  173.357905]  [<c05c591e>] ? dev_watchdog+0x0/0xec
[  173.358073]  [<c05c591e>] ? dev_watchdog+0x0/0xec
[  173.360804]  [<c0129ad7>] __do_softirq+0xb2/0x15c
[  173.360804]  [<c0129a25>] ? __do_softirq+0x0/0x15c
[  173.360804]  [<c0105526>] do_softirq+0x84/0xe9
[  173.360804]  [<c0129996>] irq_exit+0x4b/0x88
[  173.360804]  [<c010ec7a>] smp_apic_timer_interrupt+0x73/0x81
[  173.360804]  [<c0103ddd>] apic_timer_interrupt+0x2d/0x34
[  173.360804]  =======================
[  173.360804] ---[ end trace a7919e7f17c0a725 ]---
[  173.396182] evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0
[  173.446150] evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0

[-- Attachment #2: config --]
[-- Type: text/plain, Size: 46021 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.26-rc6
# Fri Jun 13 13:36:47 2008
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
# CONFIG_GENERIC_LOCKBREAK is not set
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
# CONFIG_GENERIC_GPIO is not set
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_AOUT=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
# CONFIG_BOOTPARAM_SUPPORT is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# General setup
#
CONFIG_EXPERIMENTAL=y
# CONFIG_BROKEN_BOOT_ALLOWED3 is not set
# CONFIG_BROKEN_BOOT_EUROPE is not set
# CONFIG_BROKEN_BOOT_TITAN is not set
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
# CONFIG_TASKSTATS is not set
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=20
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_NS=y
# CONFIG_CGROUP_DEVICE is not set
# CONFIG_CPUSETS is not set
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
# CONFIG_CGROUP_CPUACCT is not set
CONFIG_RESOURCE_COUNTERS=y
CONFIG_MM_OWNER=y
CONFIG_CGROUP_MEM_RES_CTLR=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_COMPAT_BRK=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_MARKERS=y
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_HAVE_DMA_ATTRS is not set
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_HAVE_IMMEDIATE=y
CONFIG_IMMEDIATE=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
# CONFIG_MODULES is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_LBD is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_LSF=y
CONFIG_BLK_DEV_BSG=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_AS=y
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="anticipatory"
CONFIG_CLASSIC_RCU=y

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
# CONFIG_HIGH_RES_TIMERS is not set
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP_SUPPORT=y
CONFIG_UP_WANTED_1=y
# CONFIG_UP_WANTED_2 is not set
CONFIG_SMP=y
# CONFIG_X86_PC is not set
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_VISWS is not set
CONFIG_X86_GENERICARCH=y
CONFIG_X86_BIGSMP=y
# CONFIG_X86_RDC321X is not set
# CONFIG_X86_VSMP is not set
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
CONFIG_X86_CYCLONE_TIMER=y
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
CONFIG_MPENTIUMIII=y
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=6
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_X86_DS=y
# CONFIG_X86_PTRACE_BTS is not set
# CONFIG_HPET_TIMER is not set
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
CONFIG_NR_CPUS=32
# CONFIG_SCHED_SMT is not set
# CONFIG_SCHED_MC is not set
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_MCE is not set
CONFIG_VM86=y
CONFIG_TOSHIBA=y
CONFIG_I8K=y
# CONFIG_X86_REBOOTFIXUPS is not set
CONFIG_MICROCODE=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
# CONFIG_X86_CPUID is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
CONFIG_ILLEGAL_POINTER_VALUE=0
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
# CONFIG_SPARSEMEM_VMEMMAP_ENABLE is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_RESOURCES_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_HIGHPTE=y
CONFIG_MATH_EMULATION=y
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
# CONFIG_X86_PAT is not set
# CONFIG_EFI is not set
CONFIG_IRQBALANCE=y
# CONFIG_SECCOMP is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_SCHED_HRTICK is not set
CONFIG_KEXEC=y
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
CONFIG_ACPI=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
# CONFIG_ACPI_AC is not set
CONFIG_ACPI_BATTERY=y
# CONFIG_ACPI_BUTTON is not set
CONFIG_ACPI_VIDEO=y
# CONFIG_ACPI_FAN is not set
# CONFIG_ACPI_DOCK is not set
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
# CONFIG_ACPI_THERMAL is not set
# CONFIG_ACPI_WMI is not set
CONFIG_ACPI_ASUS=y
# CONFIG_ACPI_TOSHIBA is not set
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
# CONFIG_CPU_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
CONFIG_PCI_GODIRECT=y
# CONFIG_PCI_GOOLPC is not set
# CONFIG_PCI_GOANY is not set
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
CONFIG_EISA=y
# CONFIG_EISA_PCI_EISA is not set
# CONFIG_EISA_NAMES is not set
# CONFIG_MCA is not set
CONFIG_SCx200=y
CONFIG_SCx200HR_TIMER=y
CONFIG_OLPC=y
CONFIG_K8_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA_DEBUG=y
CONFIG_PCMCIA=y
CONFIG_PCMCIA_LOAD_CIS=y
# CONFIG_PCMCIA_IOCTL is not set
CONFIG_CARDBUS=y

#
# PC-card bridges
#
# CONFIG_YENTA is not set
CONFIG_PD6729=y
# CONFIG_I82092 is not set
# CONFIG_I82365 is not set
# CONFIG_TCIC is not set
CONFIG_PCMCIA_PROBE=y
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_FAKE=y
CONFIG_HOTPLUG_PCI_COMPAQ=y
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
CONFIG_HOTPLUG_PCI_IBM=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=y
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=y

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_AOUT is not set
# CONFIG_BINFMT_MISC is not set

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
# CONFIG_XFRM_USER is not set
CONFIG_XFRM_SUB_POLICY=y
# CONFIG_XFRM_MIGRATE is not set
CONFIG_XFRM_STATISTICS=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
# CONFIG_IP_PNP_BOOTP is not set
# CONFIG_IP_PNP_RARP is not set
CONFIG_NET_IPIP=y
CONFIG_NET_IPGRE=y
CONFIG_ARPD=y
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
CONFIG_INET_TUNNEL=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IP_VS is not set
CONFIG_IPV6=y
# CONFIG_IPV6_PRIVACY is not set
CONFIG_IPV6_ROUTER_PREF=y
# CONFIG_IPV6_ROUTE_INFO is not set
CONFIG_IPV6_OPTIMISTIC_DAD=y
# CONFIG_INET6_AH is not set
CONFIG_INET6_ESP=y
CONFIG_INET6_IPCOMP=y
CONFIG_IPV6_MIP6=y
CONFIG_INET6_XFRM_TUNNEL=y
CONFIG_INET6_TUNNEL=y
# CONFIG_INET6_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET6_XFRM_MODE_TUNNEL is not set
CONFIG_INET6_XFRM_MODE_BEET=y
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
CONFIG_IPV6_SIT=y
CONFIG_IPV6_NDISC_NODETYPE=y
# CONFIG_IPV6_TUNNEL is not set
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_PIMSM_V2=y
# CONFIG_NETLABEL is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NETFILTER=y
CONFIG_NETFILTER_DEBUG=y
# CONFIG_NETFILTER_ADVANCED is not set

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CONNTRACK=y
# CONFIG_NF_CONNTRACK_FTP is not set
# CONFIG_NF_CONNTRACK_IRC is not set
# CONFIG_NF_CONNTRACK_SIP is not set
# CONFIG_NF_CT_NETLINK is not set
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
# CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
# CONFIG_NETFILTER_XT_TARGET_TCPMSS is not set
# CONFIG_NETFILTER_XT_MATCH_CONNTRACK is not set
CONFIG_NETFILTER_XT_MATCH_MARK=y
# CONFIG_NETFILTER_XT_MATCH_POLICY is not set
# CONFIG_NETFILTER_XT_MATCH_STATE is not set

#
# IP: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
# CONFIG_IP_NF_IPTABLES is not set

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=y
CONFIG_IP6_NF_IPTABLES=y
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
# CONFIG_IP6_NF_FILTER is not set
# CONFIG_IP6_NF_MANGLE is not set
CONFIG_IP_DCCP=y
CONFIG_INET_DCCP_DIAG=y
CONFIG_IP_DCCP_ACKVEC=y

#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
CONFIG_IP_DCCP_CCID2=y
CONFIG_IP_DCCP_CCID2_DEBUG=y
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_CCID3_RTO=100
CONFIG_IP_DCCP_TFRC_LIB=y

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
CONFIG_IP_SCTP=y
CONFIG_SCTP_DBG_MSG=y
# CONFIG_SCTP_DBG_OBJCNT is not set
CONFIG_SCTP_HMAC_NONE=y
# CONFIG_SCTP_HMAC_SHA1 is not set
# CONFIG_SCTP_HMAC_MD5 is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
CONFIG_DECNET=y
CONFIG_DECNET_ROUTER=y
CONFIG_LLC=y
# CONFIG_LLC2 is not set
CONFIG_IPX=y
# CONFIG_IPX_INTERN is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
CONFIG_ECONET=y
# CONFIG_ECONET_AUNUDP is not set
# CONFIG_ECONET_NATIVE is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_SCHED is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
CONFIG_HAMRADIO=y

#
# Packet Radio protocols
#
# CONFIG_AX25 is not set
CONFIG_CAN=y
CONFIG_CAN_RAW=y
CONFIG_CAN_BCM=y

#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=y
# CONFIG_CAN_DEBUG_DEVICES is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y

#
# Wireless
#
CONFIG_CFG80211=y
CONFIG_NL80211=y
CONFIG_WIRELESS_EXT=y
# CONFIG_MAC80211 is not set
# CONFIG_IEEE80211 is not set
# CONFIG_RFKILL is not set
CONFIG_NET_9P=y
# CONFIG_NET_9P_DEBUG is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
# CONFIG_PROC_EVENTS is not set
CONFIG_PARPORT=y
# CONFIG_PARPORT_PC is not set
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_AX88796=y
# CONFIG_PARPORT_1284 is not set
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
# CONFIG_ISAPNP is not set
CONFIG_PNPBIOS=y
# CONFIG_PNPBIOS_PROC_FS is not set
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
CONFIG_BLK_CPQ_DA=y
CONFIG_BLK_CPQ_CISS_DA=y
CONFIG_CISS_SCSI_TAPE=y
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
CONFIG_BLK_DEV_NBD=y
CONFIG_BLK_DEV_SX8=y
CONFIG_BLK_DEV_UB=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
CONFIG_ATA_OVER_ETH=y
# CONFIG_MISC_DEVICES is not set
CONFIG_TIFM_CORE=y
CONFIG_HAVE_IDE=y

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=y
CONFIG_SCSI_NETLINK=y
# CONFIG_SCSI_PROC_FS is not set

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=y
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=y
# CONFIG_SCSI_FC_TGT_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=y
# CONFIG_SCSI_SAS_ATTRS is not set
CONFIG_SCSI_SRP_ATTRS=y
CONFIG_SCSI_SRP_TGT_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
CONFIG_BLK_DEV_3W_XXXX_RAID=y
CONFIG_SCSI_3W_9XXX=y
CONFIG_SCSI_7000FASST=y
CONFIG_SCSI_ACARD=y
CONFIG_SCSI_AHA152X=y
CONFIG_SCSI_AHA1542=y
# CONFIG_SCSI_AHA1740 is not set
CONFIG_SCSI_AACRAID=y
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=5000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
CONFIG_SCSI_AIC79XX=y
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=5000
CONFIG_AIC79XX_DEBUG_ENABLE=y
CONFIG_AIC79XX_DEBUG_MASK=0
CONFIG_AIC79XX_REG_PRETTY_PRINT=y
CONFIG_SCSI_DPT_I2O=y
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
CONFIG_SCSI_ARCMSR=y
CONFIG_MEGARAID_NEWGEN=y
# CONFIG_MEGARAID_MM is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_MEGARAID_SAS=y
# CONFIG_SCSI_HPTIOP is not set
CONFIG_SCSI_BUSLOGIC=y
# CONFIG_SCSI_FLASHPOINT is not set
CONFIG_SCSI_DMX3191D=y
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
CONFIG_SCSI_GENERIC_NCR5380_MMIO=y
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
CONFIG_SCSI_INIA100=y
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
CONFIG_SCSI_IPR=y
CONFIG_SCSI_IPR_TRACE=y
CONFIG_SCSI_IPR_DUMP=y
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
CONFIG_SCSI_QLOGIC_1280=y
CONFIG_SCSI_QLA_FC=y
CONFIG_SCSI_QLA_ISCSI=y
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_SIM710 is not set
CONFIG_SCSI_SYM53C416=y
# CONFIG_SCSI_DC395x is not set
CONFIG_SCSI_DC390T=y
CONFIG_SCSI_T128=y
CONFIG_SCSI_U14_34F=y
CONFIG_SCSI_U14_34F_TAGGED_QUEUE=y
# CONFIG_SCSI_U14_34F_LINKED_COMMANDS is not set
CONFIG_SCSI_U14_34F_MAX_TAGS=8
# CONFIG_SCSI_ULTRASTOR is not set
CONFIG_SCSI_NSP32=y
CONFIG_SCSI_SRP=y
# CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
# CONFIG_ATA_ACPI is not set
# CONFIG_SATA_PMP is not set
CONFIG_SATA_AHCI=y
CONFIG_SATA_SIL24=y
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
CONFIG_SATA_SIL=y
# CONFIG_SATA_SIS is not set
CONFIG_SATA_ULI=y
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
CONFIG_SATA_INIC162X=y
CONFIG_PATA_ALI=y
CONFIG_PATA_AMD=y
CONFIG_PATA_ARTOP=y
CONFIG_PATA_ATIIXP=y
# CONFIG_PATA_CMD640_PCI is not set
CONFIG_PATA_CMD64X=y
CONFIG_PATA_CS5520=y
CONFIG_PATA_CS5530=y
# CONFIG_PATA_CS5535 is not set
CONFIG_PATA_CS5536=y
# CONFIG_PATA_CYPRESS is not set
CONFIG_PATA_EFAR=y
# CONFIG_ATA_GENERIC is not set
CONFIG_PATA_HPT366=y
CONFIG_PATA_HPT37X=y
# CONFIG_PATA_HPT3X2N is not set
CONFIG_PATA_HPT3X3=y
CONFIG_PATA_HPT3X3_DMA=y
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_JMICRON=y
CONFIG_PATA_LEGACY=y
CONFIG_PATA_TRIFLEX=y
CONFIG_PATA_MARVELL=y
# CONFIG_PATA_MPIIX is not set
CONFIG_PATA_OLDPIIX=y
# CONFIG_PATA_NETCELL is not set
CONFIG_PATA_NINJA32=y
# CONFIG_PATA_NS87410 is not set
CONFIG_PATA_NS87415=y
# CONFIG_PATA_OPTI is not set
CONFIG_PATA_OPTIDMA=y
CONFIG_PATA_PCMCIA=y
CONFIG_PATA_PDC_OLD=y
# CONFIG_PATA_QDI is not set
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RZ1000=y
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
CONFIG_PATA_SIL680=y
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
CONFIG_PATA_WINBOND=y
CONFIG_PATA_WINBOND_VLB=y
CONFIG_PATA_SCH=y
# CONFIG_MD is not set
CONFIG_FUSION=y
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
# CONFIG_FUSION_SAS is not set
CONFIG_FUSION_MAX_SGE=128
# CONFIG_FUSION_LOGGING is not set

#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=y
CONFIG_FIREWIRE_OHCI=y
CONFIG_FIREWIRE_OHCI_DEBUG=y
# CONFIG_FIREWIRE_SBP2 is not set
CONFIG_IEEE1394=y

#
# Subsystem Options
#
# CONFIG_IEEE1394_VERBOSEDEBUG is not set

#
# Controllers
#

#
# Texas Instruments PCILynx requires I2C
#
# CONFIG_IEEE1394_OHCI1394 is not set

#
# Protocols
#
# CONFIG_IEEE1394_SBP2 is not set
# CONFIG_IEEE1394_ETH1394_ROM_ENTRY is not set
# CONFIG_IEEE1394_ETH1394 is not set
CONFIG_IEEE1394_RAWIO=y
# CONFIG_I2O is not set
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
CONFIG_DUMMY=y
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
CONFIG_VETH=y
CONFIG_NET_SB1000=y
CONFIG_ARCNET=y
CONFIG_ARCNET_1201=y
# CONFIG_ARCNET_1051 is not set
# CONFIG_ARCNET_RAW is not set
# CONFIG_ARCNET_CAP is not set
CONFIG_ARCNET_COM90xx=y
# CONFIG_ARCNET_COM90xxIO is not set
CONFIG_ARCNET_RIM_I=y
# CONFIG_ARCNET_COM20020 is not set
# CONFIG_PHYLIB is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
CONFIG_CASSINI=y
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_ENC28J60 is not set
# CONFIG_NET_VENDOR_RACAL is not set
CONFIG_NET_TULIP=y
# CONFIG_DE2104X is not set
# CONFIG_TULIP is not set
# CONFIG_DE4X5 is not set
# CONFIG_WINBOND_840 is not set
CONFIG_DM9102=y
# CONFIG_ULI526X is not set
CONFIG_PCMCIA_XIRCOM=y
# CONFIG_AT1700 is not set
CONFIG_DEPCA=y
CONFIG_HP100=y
# CONFIG_NET_ISA is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
CONFIG_ADAPTEC_STARFIRE=y
CONFIG_ADAPTEC_STARFIRE_NAPI=y
CONFIG_AC3200=y
# CONFIG_APRICOT is not set
CONFIG_B44=y
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_FORCEDETH=y
# CONFIG_FORCEDETH_NAPI is not set
# CONFIG_CS89x0 is not set
CONFIG_EEPRO100=y
CONFIG_E100=y
# CONFIG_LNE390 is not set
CONFIG_FEALNX=y
CONFIG_NATSEMI=y
CONFIG_NE2K_PCI=y
CONFIG_NE3210=y
# CONFIG_ES3210 is not set
CONFIG_8139CP=y
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
CONFIG_8139_OLD_RX_RESET=y
# CONFIG_R6040 is not set
CONFIG_SIS900=y
# CONFIG_EPIC100 is not set
CONFIG_SUNDANCE=y
CONFIG_SUNDANCE_MMIO=y
CONFIG_TLAN=y
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
CONFIG_NET_POCKET=y
# CONFIG_ATP is not set
CONFIG_DE600=y
# CONFIG_DE620 is not set
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
CONFIG_E1000=y
CONFIG_E1000_NAPI=y
CONFIG_E1000_DISABLE_PACKET_SPLIT=y
# CONFIG_E1000E_ENABLED is not set
CONFIG_IP1000=y
CONFIG_IGB=y
CONFIG_NS83820=y
CONFIG_HAMACHI=y
CONFIG_YELLOWFIN=y
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
CONFIG_VIA_VELOCITY=y
CONFIG_TIGON3=y
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set
CONFIG_ATL1=y
CONFIG_NETDEV_10000=y
# CONFIG_CHELSIO_T1 is not set
# CONFIG_CHELSIO_T3 is not set
CONFIG_IXGBE=y
CONFIG_IXGB=y
# CONFIG_IXGB_NAPI is not set
# CONFIG_S2IO is not set
CONFIG_MYRI10GE=y
CONFIG_NETXEN_NIC=y
CONFIG_NIU=y
# CONFIG_MLX4_CORE is not set
CONFIG_TEHUTI=y
CONFIG_BNX2X=y
CONFIG_SFC=y
CONFIG_TR=y
# CONFIG_IBMTR is not set
CONFIG_IBMOL=y
# CONFIG_IBMLS is not set
CONFIG_3C359=y
# CONFIG_TMS380TR is not set
# CONFIG_SMCTR is not set

#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_IWLWIFI_LEDS is not set

#
# USB Network Adapters
#
CONFIG_USB_CATC=y
# CONFIG_USB_KAWETH is not set
CONFIG_USB_PEGASUS=y
# CONFIG_USB_RTL8150 is not set
CONFIG_USB_USBNET=y
CONFIG_USB_NET_AX8817X=y
# CONFIG_USB_NET_CDCETHER is not set
# CONFIG_USB_NET_DM9601 is not set
CONFIG_USB_NET_GL620A=y
CONFIG_USB_NET_NET1080=y
CONFIG_USB_NET_PLUSB=y
# CONFIG_USB_NET_MCS7830 is not set
# CONFIG_USB_NET_RNDIS_HOST is not set
CONFIG_USB_NET_CDC_SUBSET=y
# CONFIG_USB_ALI_M5632 is not set
# CONFIG_USB_AN2720 is not set
CONFIG_USB_BELKIN=y
# CONFIG_USB_ARMLINUX is not set
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
# CONFIG_USB_NET_ZAURUS is not set
# CONFIG_NET_PCMCIA is not set
# CONFIG_WAN is not set
CONFIG_FDDI=y
CONFIG_DEFXX=y
# CONFIG_DEFXX_MMIO is not set
# CONFIG_SKFP is not set
CONFIG_HIPPI=y
CONFIG_ROADRUNNER=y
CONFIG_ROADRUNNER_LARGE_RINGS=y
# CONFIG_PLIP is not set
CONFIG_PPP=y
CONFIG_PPP_MULTILINK=y
# CONFIG_PPP_FILTER is not set
CONFIG_PPP_ASYNC=y
# CONFIG_PPP_SYNC_TTY is not set
# CONFIG_PPP_DEFLATE is not set
# CONFIG_PPP_BSDCOMP is not set
CONFIG_PPP_MPPE=y
# CONFIG_PPPOE is not set
CONFIG_PPPOL2TP=y
CONFIG_SLIP=y
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=y
# CONFIG_SLIP_SMART is not set
# CONFIG_SLIP_MODE_SLIP6 is not set
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=y
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
CONFIG_INPUT_POLLDEV=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=y
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
CONFIG_KEYBOARD_XTKBD=y
CONFIG_KEYBOARD_NEWTON=y
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_INPUT_MOUSE is not set
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=y
CONFIG_JOYSTICK_A3D=y
# CONFIG_JOYSTICK_ADI is not set
# CONFIG_JOYSTICK_COBRA is not set
# CONFIG_JOYSTICK_GF2K is not set
# CONFIG_JOYSTICK_GRIP is not set
# CONFIG_JOYSTICK_GRIP_MP is not set
# CONFIG_JOYSTICK_GUILLEMOT is not set
# CONFIG_JOYSTICK_INTERACT is not set
# CONFIG_JOYSTICK_SIDEWINDER is not set
# CONFIG_JOYSTICK_TMDC is not set
CONFIG_JOYSTICK_IFORCE=y
# CONFIG_JOYSTICK_IFORCE_USB is not set
# CONFIG_JOYSTICK_IFORCE_232 is not set
CONFIG_JOYSTICK_WARRIOR=y
# CONFIG_JOYSTICK_MAGELLAN is not set
CONFIG_JOYSTICK_SPACEORB=y
# CONFIG_JOYSTICK_SPACEBALL is not set
CONFIG_JOYSTICK_STINGER=y
CONFIG_JOYSTICK_TWIDJOY=y
CONFIG_JOYSTICK_ZHENHUA=y
# CONFIG_JOYSTICK_DB9 is not set
CONFIG_JOYSTICK_GAMECON=y
CONFIG_JOYSTICK_TURBOGRAFX=y
CONFIG_JOYSTICK_JOYDUMP=y
CONFIG_JOYSTICK_XPAD=y
# CONFIG_JOYSTICK_XPAD_FF is not set
# CONFIG_JOYSTICK_XPAD_LEDS is not set
# CONFIG_INPUT_TABLET is not set
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_ADS7846=y
# CONFIG_TOUCHSCREEN_FUJITSU is not set
# CONFIG_TOUCHSCREEN_GUNZE is not set
CONFIG_TOUCHSCREEN_ELO=y
CONFIG_TOUCHSCREEN_MTOUCH=y
CONFIG_TOUCHSCREEN_MK712=y
CONFIG_TOUCHSCREEN_PENMOUNT=y
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
CONFIG_TOUCHSCREEN_TOUCHWIN=y
# CONFIG_TOUCHSCREEN_UCB1400 is not set
CONFIG_TOUCHSCREEN_USB_COMPOSITE=y
CONFIG_TOUCHSCREEN_USB_EGALAX=y
CONFIG_TOUCHSCREEN_USB_PANJIT=y
CONFIG_TOUCHSCREEN_USB_3M=y
CONFIG_TOUCHSCREEN_USB_ITM=y
CONFIG_TOUCHSCREEN_USB_ETURBO=y
CONFIG_TOUCHSCREEN_USB_GUNZE=y
CONFIG_TOUCHSCREEN_USB_DMC_TSC10=y
CONFIG_TOUCHSCREEN_USB_IRTOUCH=y
CONFIG_TOUCHSCREEN_USB_IDEALTEK=y
CONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=y
CONFIG_TOUCHSCREEN_USB_GOTOP=y
# CONFIG_INPUT_MISC is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
CONFIG_SERIO_CT82C710=y
CONFIG_SERIO_PARKBD=y
CONFIG_SERIO_PCIPS2=y
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=y
CONFIG_GAMEPORT=y
# CONFIG_GAMEPORT_NS558 is not set
# CONFIG_GAMEPORT_L4 is not set
CONFIG_GAMEPORT_EMU10K1=y
CONFIG_GAMEPORT_FM801=y

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_DEVKMEM=y
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_COMPUTONE=y
CONFIG_ROCKETPORT=y
CONFIG_CYCLADES=y
CONFIG_CYZ_INTR=y
CONFIG_DIGIEPCA=y
# CONFIG_ESPSERIAL is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
CONFIG_ISI=y
CONFIG_SYNCLINK=y
CONFIG_SYNCLINKMP=y
CONFIG_SYNCLINK_GT=y
# CONFIG_N_HDLC is not set
CONFIG_RISCOM8=y
# CONFIG_SPECIALIX is not set
CONFIG_SX=y
CONFIG_RIO=y
CONFIG_RIO_OLDPCI=y
CONFIG_STALDRV=y
CONFIG_NOZOMI=y

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CS=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
# CONFIG_PRINTER is not set
CONFIG_PPDEV=y
CONFIG_IPMI_HANDLER=y
CONFIG_IPMI_PANIC_EVENT=y
# CONFIG_IPMI_PANIC_STRING is not set
# CONFIG_IPMI_DEVICE_INTERFACE is not set
# CONFIG_IPMI_SI is not set
CONFIG_IPMI_WATCHDOG=y
CONFIG_IPMI_POWEROFF=y
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
CONFIG_DTLK=y
CONFIG_R3964=y
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# PCMCIA character devices
#
CONFIG_SYNCLINK_CS=y
CONFIG_CARDMAN_4000=y
# CONFIG_CARDMAN_4040 is not set
CONFIG_IPWIRELESS=y
# CONFIG_MWAVE is not set
# CONFIG_SCx200_GPIO is not set
CONFIG_PC8736x_GPIO=y
CONFIG_NSC_GPIO=y
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
# CONFIG_HANGCHECK_TIMER is not set
CONFIG_TCG_TPM=y
CONFIG_TCG_TIS=y
# CONFIG_TCG_NSC is not set
CONFIG_TCG_ATMEL=y
CONFIG_TCG_INFINEON=y
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_I2C is not set
CONFIG_SPI=y
CONFIG_SPI_DEBUG=y
CONFIG_SPI_MASTER=y

#
# SPI Master Controller Drivers
#
# CONFIG_SPI_BITBANG is not set
# CONFIG_SPI_BUTTERFLY is not set
# CONFIG_SPI_LM70_LLP is not set

#
# SPI Protocol Masters
#
# CONFIG_SPI_AT25 is not set
# CONFIG_SPI_SPIDEV is not set
# CONFIG_SPI_TLE62X0 is not set
CONFIG_W1=y
CONFIG_W1_CON=y

#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=y
CONFIG_W1_MASTER_DS2490=y

#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=y
CONFIG_W1_SLAVE_SMEM=y
# CONFIG_W1_SLAVE_DS2433 is not set
CONFIG_W1_SLAVE_DS2760=y
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
CONFIG_PDA_POWER=y
CONFIG_BATTERY_DS2760=y
CONFIG_BATTERY_OLPC=y
# CONFIG_HWMON is not set
CONFIG_THERMAL=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_NOWAYOUT=y

#
# Watchdog Device Drivers
#
# CONFIG_SOFT_WATCHDOG is not set
CONFIG_ACQUIRE_WDT=y
# CONFIG_ADVANTECH_WDT is not set
CONFIG_ALIM1535_WDT=y
CONFIG_ALIM7101_WDT=y
# CONFIG_SC520_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
# CONFIG_WAFER_WDT is not set
CONFIG_I6300ESB_WDT=y
# CONFIG_ITCO_WDT is not set
# CONFIG_IT8712F_WDT is not set
CONFIG_HP_WATCHDOG=y
CONFIG_SC1200_WDT=y
CONFIG_SCx200_WDT=y
# CONFIG_PC87413_WDT is not set
CONFIG_60XX_WDT=y
# CONFIG_SBC8360_WDT is not set
CONFIG_SBC7240_WDT=y
CONFIG_CPU5_WDT=y
CONFIG_SMSC37B787_WDT=y
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83697HF_WDT is not set
CONFIG_W83877F_WDT=y
CONFIG_W83977F_WDT=y
# CONFIG_MACHZ_WDT is not set
# CONFIG_SBC_EPX_C3_WATCHDOG is not set

#
# ISA-based Watchdog Cards
#
CONFIG_PCWATCHDOG=y
CONFIG_MIXCOMWD=y

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
CONFIG_WDTPCI=y
# CONFIG_WDT_501_PCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set

#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
CONFIG_SSB=y
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
CONFIG_SSB_PCMCIAHOST=y
CONFIG_SSB_DEBUG=y
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y

#
# Multifunction device drivers
#
CONFIG_MFD_SM501=y
CONFIG_HTC_PASIC3=y

#
# Multimedia devices
#

#
# Multimedia core support
#
# CONFIG_VIDEO_DEV is not set
CONFIG_DVB_CORE=y
CONFIG_VIDEO_MEDIA=y

#
# Multimedia drivers
#
# CONFIG_DVB_CAPTURE_DRIVERS is not set
# CONFIG_DAB is not set

#
# Graphics support
#
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
CONFIG_AGP_ATI=y
CONFIG_AGP_AMD=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL is not set
CONFIG_AGP_NVIDIA=y
# CONFIG_AGP_SIS is not set
CONFIG_AGP_SWORKS=y
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=y
# CONFIG_FB is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_CORGI=y
# CONFIG_BACKLIGHT_PROGEAR is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
# CONFIG_VIDEO_SELECT is not set
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
CONFIG_SOUND=y

#
# Advanced Linux Sound Architecture
#
# CONFIG_SND is not set

#
# Open Sound System
#
CONFIG_SOUND_PRIME=y
# CONFIG_SOUND_TRIDENT is not set
# CONFIG_HID_SUPPORT is not set
CONFIG_USB_MOUSE=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
CONFIG_USB_DEBUG=y
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
CONFIG_USB_DEVICE_CLASS=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=y
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
CONFIG_USB_ISP116X_HCD=y
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_SSB=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_SL811_HCD=y
# CONFIG_USB_SL811_CS is not set
# CONFIG_USB_R8A66597_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=y
CONFIG_USB_WDM=y

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#

#
# may also be needed; see USB_STORAGE Help for more information
#
# CONFIG_USB_STORAGE is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_MON is not set

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
CONFIG_USB_EMI26=y
# CONFIG_USB_ADUTUX is not set
CONFIG_USB_AUERSWALD=y
CONFIG_USB_RIO500=y
CONFIG_USB_LEGOTOWER=y
CONFIG_USB_LCD=y
CONFIG_USB_BERRY_CHARGE=y
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
CONFIG_USB_CYTHERM=y
# CONFIG_USB_PHIDGET is not set
CONFIG_USB_IDMOUSE=y
# CONFIG_USB_FTDI_ELAN is not set
CONFIG_USB_APPLEDISPLAY=y
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
CONFIG_USB_ISIGHTFW=y
CONFIG_USB_GADGET=y
CONFIG_USB_GADGET_DEBUG=y
CONFIG_USB_GADGET_DEBUG_FILES=y
# CONFIG_USB_GADGET_DEBUG_FS is not set
CONFIG_USB_GADGET_SELECTED=y
# CONFIG_USB_GADGET_AMD5536UDC is not set
# CONFIG_USB_GADGET_ATMEL_USBA is not set
# CONFIG_USB_GADGET_FSL_USB2 is not set
# CONFIG_USB_GADGET_NET2280 is not set
# CONFIG_USB_GADGET_PXA2XX is not set
# CONFIG_USB_GADGET_M66592 is not set
# CONFIG_USB_GADGET_PXA27X is not set
# CONFIG_USB_GADGET_GOKU is not set
# CONFIG_USB_GADGET_LH7A40X is not set
# CONFIG_USB_GADGET_OMAP is not set
# CONFIG_USB_GADGET_S3C2410 is not set
# CONFIG_USB_GADGET_AT91 is not set
CONFIG_USB_GADGET_DUMMY_HCD=y
CONFIG_USB_DUMMY_HCD=y
CONFIG_USB_GADGET_DUALSPEED=y
# CONFIG_USB_ZERO is not set
# CONFIG_USB_ETH is not set
CONFIG_USB_GADGETFS=y
# CONFIG_USB_FILE_STORAGE is not set
# CONFIG_USB_G_SERIAL is not set
# CONFIG_USB_MIDI_GADGET is not set
# CONFIG_USB_G_PRINTER is not set
# CONFIG_MMC is not set
CONFIG_MEMSTICK=y
# CONFIG_MEMSTICK_DEBUG is not set

#
# MemoryStick drivers
#
CONFIG_MEMSTICK_UNSAFE_RESUME=y
CONFIG_MSPRO_BLOCK=y

#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=y
CONFIG_MEMSTICK_JMICRON_38X=y
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

#
# LED drivers
#
# CONFIG_LEDS_CLEVO_MAIL is not set

#
# LED Triggers
#
# CONFIG_LEDS_TRIGGERS is not set
CONFIG_ACCESSIBILITY=y
CONFIG_A11Y_BRAILLE_CONSOLE=y
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y

#
# Reporting subsystems
#
CONFIG_EDAC_DEBUG=y
# CONFIG_EDAC_MM_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
# CONFIG_RTC_HCTOSYS is not set
CONFIG_RTC_DEBUG=y

#
# RTC interfaces
#
# CONFIG_RTC_INTF_SYSFS is not set
# CONFIG_RTC_INTF_PROC is not set
# CONFIG_RTC_INTF_DEV is not set
CONFIG_RTC_DRV_TEST=y

#
# SPI RTC drivers
#
CONFIG_RTC_DRV_MAX6902=y
CONFIG_RTC_DRV_R9701=y
CONFIG_RTC_DRV_RS5C348=y

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_DS1511=y
CONFIG_RTC_DRV_DS1553=y
CONFIG_RTC_DRV_DS1742=y
# CONFIG_RTC_DRV_STK17TA8 is not set
CONFIG_RTC_DRV_M48T86=y
CONFIG_RTC_DRV_M48T59=y
# CONFIG_RTC_DRV_V3020 is not set

#
# on-CPU RTC drivers
#
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=y
CONFIG_UIO_CIF=y
CONFIG_UIO_SMX=y

#
# Firmware Drivers
#
CONFIG_EDD=y
# CONFIG_EDD_OFF is not set
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
# CONFIG_EXT2_FS_POSIX_ACL is not set
# CONFIG_EXT2_FS_SECURITY is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
# CONFIG_EXT4DEV_FS is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
CONFIG_OCFS2_FS=y
# CONFIG_OCFS2_FS_O2CB is not set
# CONFIG_OCFS2_DEBUG_MASKLOG is not set
CONFIG_OCFS2_DEBUG_FS=y
CONFIG_DNOTIFY=y
# CONFIG_INOTIFY is not set
CONFIG_QUOTA=y
# CONFIG_QUOTA_NETLINK_INTERFACE is not set
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QFMT_V1 is not set
# CONFIG_QFMT_V2 is not set
CONFIG_QUOTACTL=y
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set

#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
CONFIG_HFS_FS=y
# CONFIG_HFSPLUS_FS is not set
CONFIG_BEFS_FS=y
# CONFIG_BEFS_DEBUG is not set
CONFIG_BFS_FS=y
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=y
CONFIG_VXFS_FS=y
# CONFIG_MINIX_FS is not set
CONFIG_HPFS_FS=y
# CONFIG_QNX4FS_FS is not set
CONFIG_ROMFS_FS=y
CONFIG_SYSV_FS=y
CONFIG_UFS_FS=y
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_UFS_DEBUG is not set
# CONFIG_NETWORK_FILESYSTEMS is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
CONFIG_ACORN_PARTITION_CUMANA=y
CONFIG_ACORN_PARTITION_EESOX=y
CONFIG_ACORN_PARTITION_ICS=y
CONFIG_ACORN_PARTITION_ADFS=y
# CONFIG_ACORN_PARTITION_POWERTEC is not set
CONFIG_ACORN_PARTITION_RISCIX=y
# CONFIG_OSF_PARTITION is not set
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
CONFIG_ULTRIX_PARTITION=y
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=y
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
CONFIG_NLS_CODEPAGE_852=y
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
CONFIG_NLS_CODEPAGE_861=y
# CONFIG_NLS_CODEPAGE_862 is not set
CONFIG_NLS_CODEPAGE_863=y
# CONFIG_NLS_CODEPAGE_864 is not set
CONFIG_NLS_CODEPAGE_865=y
CONFIG_NLS_CODEPAGE_866=y
# CONFIG_NLS_CODEPAGE_869 is not set
CONFIG_NLS_CODEPAGE_936=y
# CONFIG_NLS_CODEPAGE_950 is not set
CONFIG_NLS_CODEPAGE_932=y
CONFIG_NLS_CODEPAGE_949=y
CONFIG_NLS_CODEPAGE_874=y
CONFIG_NLS_ISO8859_8=y
CONFIG_NLS_CODEPAGE_1250=y
CONFIG_NLS_CODEPAGE_1251=y
CONFIG_NLS_ASCII=y
# CONFIG_NLS_ISO8859_1 is not set
CONFIG_NLS_ISO8859_2=y
# CONFIG_NLS_ISO8859_3 is not set
CONFIG_NLS_ISO8859_4=y
# CONFIG_NLS_ISO8859_5 is not set
CONFIG_NLS_ISO8859_6=y
CONFIG_NLS_ISO8859_7=y
CONFIG_NLS_ISO8859_9=y
# CONFIG_NLS_ISO8859_13 is not set
CONFIG_NLS_ISO8859_14=y
CONFIG_NLS_ISO8859_15=y
# CONFIG_NLS_KOI8_R is not set
CONFIG_NLS_KOI8_U=y
CONFIG_NLS_UTF8=y
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_DETECT_SOFTLOCKUP is not set
# CONFIG_SCHED_DEBUG is not set
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
CONFIG_DEBUG_OBJECTS=y
CONFIG_DEBUG_OBJECTS_SELFTEST=y
CONFIG_DEBUG_OBJECTS_FREE=y
CONFIG_DEBUG_OBJECTS_TIMERS=y
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_PI_LIST=y
# CONFIG_RT_MUTEX_TESTER is not set
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
# CONFIG_PROVE_LOCKING is not set
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
# CONFIG_DEBUG_LOCKDEP is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
CONFIG_DEBUG_LOCKING_API_SELFTESTS=y
CONFIG_STACKTRACE=y
CONFIG_DEBUG_HIGHMEM=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_VM=y
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_SG=y
CONFIG_FRAME_POINTER=y
CONFIG_BOOT_PRINTK_DELAY=y
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_HAVE_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACING=y
# CONFIG_FTRACE is not set
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SYSPROF_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_CONTEXT_SWITCH_TRACER=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set
CONFIG_SAMPLES=y
CONFIG_SAMPLE_KOBJECT=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
# CONFIG_KGDB_SERIAL_CONSOLE is not set
CONFIG_KGDB_TESTS=y
CONFIG_NONPROMISC_DEVMEM=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
CONFIG_DEBUG_STACK_USAGE=y
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_X86_PTDUMP=y
# CONFIG_DEBUG_RODATA is not set
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y
CONFIG_MMIOTRACE_HOOKS=y
CONFIG_MMIOTRACE=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
# CONFIG_IO_DELAY_0X80 is not set
# CONFIG_IO_DELAY_0XED is not set
CONFIG_IO_DELAY_UDELAY=y
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=2
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_CPA_DEBUG=y

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_SECURITY_CAPABILITIES=y
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=0
# CONFIG_SECURITY_SELINUX is not set
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_GF128MUL=y
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=y

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=y
# CONFIG_CRYPTO_GCM is not set
CONFIG_CRYPTO_SEQIV=y

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=y
CONFIG_CRYPTO_ECB=y
# CONFIG_CRYPTO_LRW is not set
CONFIG_CRYPTO_PCBC=y
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=y

#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_TGR192=y
CONFIG_CRYPTO_WP512=y

#
# Ciphers
#
# CONFIG_CRYPTO_AES is not set
# CONFIG_CRYPTO_AES_586 is not set
CONFIG_CRYPTO_ANUBIS=y
CONFIG_CRYPTO_ARC4=y
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
CONFIG_CRYPTO_CAST5=y
CONFIG_CRYPTO_CAST6=y
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_FCRYPT is not set
CONFIG_CRYPTO_KHAZAD=y
CONFIG_CRYPTO_SALSA20=y
# CONFIG_CRYPTO_SALSA20_586 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
CONFIG_CRYPTO_TEA=y
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_586 is not set

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
# CONFIG_CRYPTO_LZO is not set
# CONFIG_CRYPTO_HW is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC_CCITT=y
# CONFIG_CRC16 is not set
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=y
CONFIG_AUDIT_GENERIC=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_FORCE_SUCCESSFUL_BUILD=y
CONFIG_FORCE_MINIMAL_CONFIG=y
CONFIG_FORCE_MINIMAL_CONFIG_PHYS=y
CONFIG_X86_32_ALWAYS_ON=y

[-- Attachment #3: boot.log.bz2 --]
[-- Type: application/x-bzip2, Size: 62071 bytes --]

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > far, and updated it to yours. The delta between the two is the 3 lines 
> > patch below.
> > 
> > A few testsystems already booted into your patch, so if i dont report 
> > a hung TCP connection in the next 6 hours consider it:
> > 
> > Tested-by: Ingo Molnar <mingo@elte.hu>
> 
> this threw the warning below - never saw that before in thousands of 
> bootups and this was the only networking change that happened. config 
> and bootlog attached. Might be unlucky coincidence.

hm, threw a second warning after 6 more hours of testing:

[  362.170209] WARNING: at net/sched/sch_generic.c:222 dev_watchdog+0xde/0xf0()

that appears to be more than just coincidence. I've applied the patch 
below - which brings me back to the well-tested revert from Ilpo.

This is the only change i've done for the overnight -tip testruns, so if 
the warning from sch_generic.c goes away it's this change that has an 
impact on that warning.

	Ingo

--------------------->
commit 3019ae9652fe44c099669e5dba116acad583cfcb
Author: Ingo Molnar <mingo@elte.hu>
Date:   Fri Jun 13 23:09:28 2008 +0200

    tcp: revert again
    
    Signed-off-by: Ingo Molnar <mingo@elte.hu>

diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 045e799..ec83448 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -466,9 +466,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
 		reqp=&lopt->syn_table[i];
 		while ((req = *reqp) != NULL) {
 			if (time_after_eq(now, req->expires)) {
-				if ((req->retrans < (inet_rsk(req)->acked ? max_retries : thresh)) &&
-				    (inet_rsk(req)->acked ||
-				     !req->rsk_ops->rtx_syn_ack(parent, req))) {
+				if ((req->retrans < thresh ||
+				     (inet_rsk(req)->acked && req->retrans < max_retries))
+				    && !req->rsk_ops->rtx_syn_ack(parent, req)) {
 					unsigned long timeo;
 
 					if (req->retrans++ == 0)

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Fri, 13 Jun 2008 13:47:46 +0200

> this threw the warning below - never saw that before in thousands of 
> bootups and this was the only networking change that happened. config 
> and bootlog attached. Might be unlucky coincidence.

So that we can make forward progress here, please confirm that the
following patch against -tip makes your problems go away for good.

Once you can confirm I will push it to Linus.

Thanks!

tcp: Revert reset of deferred accept changes in 2.6.26

Ingo's system is still seeing strange behavior, and he
reports that is goes away if the rest of the deferred
accept changes are reverted too.

Therefore this reverts e4c78840284f3f51b1896cf3936d60a6033c4d2c
("[TCP]: TCP_DEFER_ACCEPT updates - dont retxmt synack") and
539fae89bebd16ebeafd57a87169bc56eb530d76 ("[TCP]: TCP_DEFER_ACCEPT
updates - defer timeout conflicts with max_thresh").

Just like the other revert, these ideas can be revisited for
2.6.27

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 net/ipv4/inet_connection_sock.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 045e799..ec83448 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -466,9 +466,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
 		reqp=&lopt->syn_table[i];
 		while ((req = *reqp) != NULL) {
 			if (time_after_eq(now, req->expires)) {
-				if ((req->retrans < (inet_rsk(req)->acked ? max_retries : thresh)) &&
-				    (inet_rsk(req)->acked ||
-				     !req->rsk_ops->rtx_syn_ack(parent, req))) {
+				if ((req->retrans < thresh ||
+				     (inet_rsk(req)->acked && req->retrans < max_retries))
+				    && !req->rsk_ops->rtx_syn_ack(parent, req)) {
 					unsigned long timeo;
 
 					if (req->retrans++ == 0)
-- 
1.5.5.1.308.g1fbb5

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Fri, 13 Jun 2008 13:47:46 +0200
> 
> > this threw the warning below - never saw that before in thousands of 
> > bootups and this was the only networking change that happened. 
> > config and bootlog attached. Might be unlucky coincidence.
> 
> So that we can make forward progress here, please confirm that the 
> following patch against -tip makes your problems go away for good.
> 
> Once you can confirm I will push it to Linus.

i triggered the net/sched/sch_generic.c:222 warning once more meanwhile 
(yesterday) with the full revert applied (which i think is the same as 
the patch below).

So i think it's either some unlucky coincidence or some timing 
relationship - perhaps the change impacts packet ordering for certain 
workload patterns? [but that same condition can occur without that patch 
too]

I also checked kerneloops.org and this warning seems to have been 
reported by others as well - although it's not triggering heavily. In 
some of those other reports the warning came together with a dead 
interface, while in my case it's just a warning with still working 
networking.

So since there's no clear bug pattern and no sure reproducability on my 
side i'd suggest we track this problem separately and "do nothing" right 
now. I've excluded this warning from my 'is the freshly booted kernel 
buggy' list of conditions of -tip testing so it's not holding me up.

and i can apply any test-patch if that would be helpful - if it does a 
WARN_ON() i'll notice it. (pure extra debug printks with no stack trace 
are much harder to notice in automated tests)

btw., it would be nice if there was some .config driven networking debug 
option that randomized packet ordering in the tx and rx queue. 
(transparently enabled, with zero-config on the userspace side)

I.e. it would have an (expensive, because O(1)) debug mechanism that 
randomized things - it would insert new packets into a random place 
within the queue where it gets queued. We could hit races and rarer 
codepaths much sooner that way - as especially in LAN based testing 
there's a strong natural ordering of packets so randomizing it 
artificially looks promising to me.

If you make that new option =y enable-able in the .config(dependent on 
DEBUG_KERNEL && default off, etc.), and as long as it does not have to 
be configured on the userspace side (i'm testing unmodified userspace 
images with default distro installs, etc.) the randconfig test will 
still be able to reach it in a percentage of the tests and i think we'll 
be able to hit a lot of exciting races much sooner than with the normal 
in-order/FIFO queueing methods.

it's basically massively parallel coverage testing. It doesnt matter how 
unbelievably slow packet ordering randomization might be, the coverage 
testing it would do would be worth gold i'm sure. (I'd love to test 
something like that in -tip, if it comes in form of some standalone 
patch against a mainline-ish tree.)

	Ingo

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Tue, 17 Jun 2008 09:26:58 +0200

> So since there's no clear bug pattern and no sure reproducability on my 
> side i'd suggest we track this problem separately and "do nothing" right 
> now. I've excluded this warning from my 'is the freshly booted kernel 
> buggy' list of conditions of -tip testing so it's not holding me up.

I'm going to push the revert through just to be safe and I think it's
a good idea to do so because all of those defer accept changes should
be resubmitted as a group for 2.6.27

> and i can apply any test-patch if that would be helpful - if it does a 
> WARN_ON() i'll notice it. (pure extra debug printks with no stack trace 
> are much harder to notice in automated tests)

I don't have time to work on your bug, sorry.  Someone else will
have to step forward and help you with it.

FWIW I don't think your TX timeout problem has anything to do with
packet ordering.  The TX element of the network device is totally
stateless, but it's hanging under some set of circumstances to the
point where we timeout and reset the hardware to get it going again.

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Tue, 17 Jun 2008 09:26:58 +0200
> 
> > So since there's no clear bug pattern and no sure reproducability on 
> > my side i'd suggest we track this problem separately and "do 
> > nothing" right now. I've excluded this warning from my 'is the 
> > freshly booted kernel buggy' list of conditions of -tip testing so 
> > it's not holding me up.
> 
> I'm going to push the revert through just to be safe and I think it's 
> a good idea to do so because all of those defer accept changes should 
> be resubmitted as a group for 2.6.27

okay - in that case the full revert is well-tested on my side as well, 
fwiw.

Tested-by: Ingo Molnar <mingo@elte.hu>

> > and i can apply any test-patch if that would be helpful - if it does 
> > a WARN_ON() i'll notice it. (pure extra debug printks with no stack 
> > trace are much harder to notice in automated tests)
> 
> I don't have time to work on your bug, sorry.  Someone else will have 
> to step forward and help you with it.

it's not really "my bug" - i just offered help to debug someone else's 
bug :-) This is pretty common hw so i guess there will be such reports.

Let me describe what i'm doing exactly: i do a lot of randomized testing 
on about a dozen real systems (all across the x86 spectrum) so i tend to 
trigger a lot of mainline bugs pretty early on.

My collection of kernel bugs for the last 8 months shows 1285 bugs 
(kernel crashes or build failures - about 50%/50%) triggered. One 
test-system alone has a serial log of 15 gigabytes - and there's a dozen 
of them. That's about 5 kernel bugs a day handled by me, on average.

These systems have about 10 times the hardware variability of your 
Niagara system for example, and many of them are rather difficult to 
debug (laptops without serial port, etc.). So i physically cannot avoid 
and debug all bugs on all my test-systems, like you do on the Niagara. I 
will report bugs, i'll bisect anything that is bisectable (on average i 
bisect once a day), and i can add patches and report any test-results, 
and i'll of course debug any bugs that look like heavy mainline 
showstoppers.

> FWIW I don't think your TX timeout problem has anything to do with 
> packet ordering.  The TX element of the network device is totally 
> stateless, but it's hanging under some set of circumstances to the 
> point where we timeout and reset the hardware to get it going again.

ok. That's e1000 then. Cc:s added. Stock T60 laptop, 32-bit:

02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
        Subsystem: Lenovo ThinkPad T60
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 2000 [size=32]
        Capabilities: <access denied>
        Kernel driver in use: e1000

the problem is this non-fatal warning showing up after bootup, 
sporadically, in a non-reproducible way:

[  173.354049] NETDEV WATCHDOG: eth0: transmit timed out
[  173.354148] ------------[ cut here ]------------
[  173.354221] WARNING: at net/sched/sch_generic.c:222 dev_watchdog+0x9a/0xec()
[  173.354298] Modules linked in:
[  173.354421] Pid: 13452, comm: cc1 Tainted: G        W 2.6.26-rc6-00273-g81ae43a-dirty #2573
[  173.354516]  [<c01250ca>] warn_on_slowpath+0x46/0x76
[  173.354641]  [<c011d428>] ? try_to_wake_up+0x1d6/0x1e0
[  173.354815]  [<c01411e9>] ? trace_hardirqs_off+0xb/0xd
[  173.357370]  [<c011d43d>] ? default_wake_function+0xb/0xd
[  173.357370]  [<c014112a>] ? trace_hardirqs_off_caller+0x15/0xc9
[  173.357370]  [<c01411e9>] ? trace_hardirqs_off+0xb/0xd
[  173.357370]  [<c0142c83>] ? trace_hardirqs_on+0xb/0xd
[  173.357370]  [<c0142b33>] ? trace_hardirqs_on_caller+0x16/0x15b
[  173.357370]  [<c0142c83>] ? trace_hardirqs_on+0xb/0xd
[  173.357370]  [<c06bb3c9>] ? _spin_unlock_irqrestore+0x5b/0x71
[  173.357370]  [<c0133d46>] ? __queue_work+0x2d/0x32
[  173.357370]  [<c0134023>] ? queue_work+0x50/0x72
[  173.357483]  [<c0134059>] ? schedule_work+0x14/0x16
[  173.357654]  [<c05c59b8>] dev_watchdog+0x9a/0xec
[  173.357783]  [<c012d456>] run_timer_softirq+0x13d/0x19d
[  173.357905]  [<c05c591e>] ? dev_watchdog+0x0/0xec
[  173.358073]  [<c05c591e>] ? dev_watchdog+0x0/0xec
[  173.360804]  [<c0129ad7>] __do_softirq+0xb2/0x15c
[  173.360804]  [<c0129a25>] ? __do_softirq+0x0/0x15c
[  173.360804]  [<c0105526>] do_softirq+0x84/0xe9
[  173.360804]  [<c0129996>] irq_exit+0x4b/0x88
[  173.360804]  [<c010ec7a>] smp_apic_timer_interrupt+0x73/0x81
[  173.360804]  [<c0103ddd>] apic_timer_interrupt+0x2d/0x34
[  173.360804]  =======================
[  173.360804] ---[ end trace a7919e7f17c0a725 ]---

full report can be found at:

   http://lkml.org/lkml/2008/6/13/224

i have 3 other test-systems with e1000 (with a similar CPU) which are 
_not_ showing this symptom, so this could be some model-specific e1000 
issue.

	Ingo

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> > FWIW I don't think your TX timeout problem has anything to do with 
> > packet ordering.  The TX element of the network device is totally 
> > stateless, but it's hanging under some set of circumstances to the 
> > point where we timeout and reset the hardware to get it going again.
> 
> ok. That's e1000 then. Cc:s added. Stock T60 laptop, 32-bit:
> 
> 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
>         Subsystem: Lenovo ThinkPad T60
>         Flags: bus master, fast devsel, latency 0, IRQ 16
>         Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
>         I/O ports at 2000 [size=32]
>         Capabilities: <access denied>
>         Kernel driver in use: e1000
> 
> the problem is this non-fatal warning showing up after bootup, 
> sporadically, in a non-reproducible way:
> 
> [  173.354049] NETDEV WATCHDOG: eth0: transmit timed out
> [  173.354148] ------------[ cut here ]------------
> [  173.354221] WARNING: at net/sched/sch_generic.c:222 dev_watchdog+0x9a/0xec()
> [  173.354298] Modules linked in:
> [  173.354421] Pid: 13452, comm: cc1 Tainted: G        W 2.6.26-rc6-00273-g81ae43a-dirty #2573
> [  173.354516]  [<c01250ca>] warn_on_slowpath+0x46/0x76
> [  173.354641]  [<c011d428>] ? try_to_wake_up+0x1d6/0x1e0
> [  173.354815]  [<c01411e9>] ? trace_hardirqs_off+0xb/0xd
> [  173.357370]  [<c011d43d>] ? default_wake_function+0xb/0xd
> [  173.357370]  [<c014112a>] ? trace_hardirqs_off_caller+0x15/0xc9
> [  173.357370]  [<c01411e9>] ? trace_hardirqs_off+0xb/0xd
> [  173.357370]  [<c0142c83>] ? trace_hardirqs_on+0xb/0xd
> [  173.357370]  [<c0142b33>] ? trace_hardirqs_on_caller+0x16/0x15b
> [  173.357370]  [<c0142c83>] ? trace_hardirqs_on+0xb/0xd
> [  173.357370]  [<c06bb3c9>] ? _spin_unlock_irqrestore+0x5b/0x71
> [  173.357370]  [<c0133d46>] ? __queue_work+0x2d/0x32
> [  173.357370]  [<c0134023>] ? queue_work+0x50/0x72
> [  173.357483]  [<c0134059>] ? schedule_work+0x14/0x16
> [  173.357654]  [<c05c59b8>] dev_watchdog+0x9a/0xec
> [  173.357783]  [<c012d456>] run_timer_softirq+0x13d/0x19d
> [  173.357905]  [<c05c591e>] ? dev_watchdog+0x0/0xec
> [  173.358073]  [<c05c591e>] ? dev_watchdog+0x0/0xec
> [  173.360804]  [<c0129ad7>] __do_softirq+0xb2/0x15c
> [  173.360804]  [<c0129a25>] ? __do_softirq+0x0/0x15c
> [  173.360804]  [<c0105526>] do_softirq+0x84/0xe9
> [  173.360804]  [<c0129996>] irq_exit+0x4b/0x88
> [  173.360804]  [<c010ec7a>] smp_apic_timer_interrupt+0x73/0x81
> [  173.360804]  [<c0103ddd>] apic_timer_interrupt+0x2d/0x34
> [  173.360804]  =======================
> [  173.360804] ---[ end trace a7919e7f17c0a725 ]---
> 
> full report can be found at:
> 
>    http://lkml.org/lkml/2008/6/13/224
> 
> i have 3 other test-systems with e1000 (with a similar CPU) which are 
> _not_ showing this symptom, so this could be some model-specific e1000 
> issue.

btw., this reminds me that this is the same system that has a serious 
e1000 network latency bug which i have reported more than a year ago, 
but which still does not appear to be fixed in latest mainline:

 PING europe (10.0.1.15) 56(84) bytes of data.
 64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=1.51 ms
 64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=404 ms
 64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=487 ms
 64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=296 ms
 64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=305 ms
 64 bytes from europe (10.0.1.15): icmp_seq=6 ttl=64 time=1011 ms
 64 bytes from europe (10.0.1.15): icmp_seq=7 ttl=64 time=0.209 ms
 64 bytes from europe (10.0.1.15): icmp_seq=8 ttl=64 time=763 ms
 64 bytes from europe (10.0.1.15): icmp_seq=9 ttl=64 time=1000 ms
 64 bytes from europe (10.0.1.15): icmp_seq=10 ttl=64 time=0.438 ms
 64 bytes from europe (10.0.1.15): icmp_seq=11 ttl=64 time=1000 ms
 64 bytes from europe (10.0.1.15): icmp_seq=12 ttl=64 time=0.299 ms
 ^C
 --- europe ping statistics ---
 12 packets transmitted, 12 received, 0% packet loss, time 11085ms

those up to 1000 msec delays can be 'felt' via ssh too, if this problem 
triggers then the system is almost unusable via the network. Local 
latencies are perfect so it's an e1000 problem.

	Ingo

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Vitaliy Gusev]

On 17 June 2008 12:09:58 Ingo Molnar wrote:
> * David Miller <davem@davemloft.net> wrote:
> > From: Ingo Molnar <mingo@elte.hu>
> > Date: Tue, 17 Jun 2008 09:26:58 +0200
> >
> > > So since there's no clear bug pattern and no sure reproducability on
> > > my side i'd suggest we track this problem separately and "do
> > > nothing" right now. I've excluded this warning from my 'is the
> > > freshly booted kernel buggy' list of conditions of -tip testing so
> > > it's not holding me up.
> >
> > I'm going to push the revert through just to be safe and I think it's
> > a good idea to do so because all of those defer accept changes should
> > be resubmitted as a group for 2.6.27
>
> okay - in that case the full revert is well-tested on my side as well,
> fwiw.
>
> Tested-by: Ingo Molnar <mingo@elte.hu>

Revert patch takes away problem with leak sockets.
Tested-by: Vitaliy Gusev <vgusev@openvz.org>

>
> > > and i can apply any test-patch if that would be helpful - if it does
> > > a WARN_ON() i'll notice it. (pure extra debug printks with no stack
> > > trace are much harder to notice in automated tests)
> >
> > I don't have time to work on your bug, sorry.  Someone else will have
> > to step forward and help you with it.
>
> it's not really "my bug" - i just offered help to debug someone else's
> bug :-) This is pretty common hw so i guess there will be such reports.
>
> Let me describe what i'm doing exactly: i do a lot of randomized testing
> on about a dozen real systems (all across the x86 spectrum) so i tend to
> trigger a lot of mainline bugs pretty early on.
>
> My collection of kernel bugs for the last 8 months shows 1285 bugs
> (kernel crashes or build failures - about 50%/50%) triggered. One
> test-system alone has a serial log of 15 gigabytes - and there's a dozen
> of them. That's about 5 kernel bugs a day handled by me, on average.
>
> These systems have about 10 times the hardware variability of your
> Niagara system for example, and many of them are rather difficult to
> debug (laptops without serial port, etc.). So i physically cannot avoid
> and debug all bugs on all my test-systems, like you do on the Niagara. I
> will report bugs, i'll bisect anything that is bisectable (on average i
> bisect once a day), and i can add patches and report any test-results,
> and i'll of course debug any bugs that look like heavy mainline
> showstoppers.
>
> > FWIW I don't think your TX timeout problem has anything to do with
> > packet ordering.  The TX element of the network device is totally
> > stateless, but it's hanging under some set of circumstances to the
> > point where we timeout and reset the hardware to get it going again.
>
> ok. That's e1000 then. Cc:s added. Stock T60 laptop, 32-bit:
>
> 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet
> Controller Subsystem: Lenovo ThinkPad T60
>         Flags: bus master, fast devsel, latency 0, IRQ 16
>         Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
>         I/O ports at 2000 [size=32]
>         Capabilities: <access denied>
>         Kernel driver in use: e1000
>
> the problem is this non-fatal warning showing up after bootup,
> sporadically, in a non-reproducible way:
>
> [  173.354049] NETDEV WATCHDOG: eth0: transmit timed out
> [  173.354148] ------------[ cut here ]------------
> [  173.354221] WARNING: at net/sched/sch_generic.c:222
> dev_watchdog+0x9a/0xec() [  173.354298] Modules linked in:
> [  173.354421] Pid: 13452, comm: cc1 Tainted: G        W
> 2.6.26-rc6-00273-g81ae43a-dirty #2573 [  173.354516]  [<c01250ca>]
> warn_on_slowpath+0x46/0x76
> [  173.354641]  [<c011d428>] ? try_to_wake_up+0x1d6/0x1e0
> [  173.354815]  [<c01411e9>] ? trace_hardirqs_off+0xb/0xd
> [  173.357370]  [<c011d43d>] ? default_wake_function+0xb/0xd
> [  173.357370]  [<c014112a>] ? trace_hardirqs_off_caller+0x15/0xc9
> [  173.357370]  [<c01411e9>] ? trace_hardirqs_off+0xb/0xd
> [  173.357370]  [<c0142c83>] ? trace_hardirqs_on+0xb/0xd
> [  173.357370]  [<c0142b33>] ? trace_hardirqs_on_caller+0x16/0x15b
> [  173.357370]  [<c0142c83>] ? trace_hardirqs_on+0xb/0xd
> [  173.357370]  [<c06bb3c9>] ? _spin_unlock_irqrestore+0x5b/0x71
> [  173.357370]  [<c0133d46>] ? __queue_work+0x2d/0x32
> [  173.357370]  [<c0134023>] ? queue_work+0x50/0x72
> [  173.357483]  [<c0134059>] ? schedule_work+0x14/0x16
> [  173.357654]  [<c05c59b8>] dev_watchdog+0x9a/0xec
> [  173.357783]  [<c012d456>] run_timer_softirq+0x13d/0x19d
> [  173.357905]  [<c05c591e>] ? dev_watchdog+0x0/0xec
> [  173.358073]  [<c05c591e>] ? dev_watchdog+0x0/0xec
> [  173.360804]  [<c0129ad7>] __do_softirq+0xb2/0x15c
> [  173.360804]  [<c0129a25>] ? __do_softirq+0x0/0x15c
> [  173.360804]  [<c0105526>] do_softirq+0x84/0xe9
> [  173.360804]  [<c0129996>] irq_exit+0x4b/0x88
> [  173.360804]  [<c010ec7a>] smp_apic_timer_interrupt+0x73/0x81
> [  173.360804]  [<c0103ddd>] apic_timer_interrupt+0x2d/0x34
> [  173.360804]  =======================
> [  173.360804] ---[ end trace a7919e7f17c0a725 ]---
>
> full report can be found at:
>
>    http://lkml.org/lkml/2008/6/13/224
>
> i have 3 other test-systems with e1000 (with a similar CPU) which are
> _not_ showing this symptom, so this could be some model-specific e1000
> issue.
>
> 	Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Thank,
Vitaliy Gusev

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Tue, 17 Jun 2008 10:32:20 +0200

> those up to 1000 msec delays can be 'felt' via ssh too, if this problem 
> triggers then the system is almost unusable via the network. Local 
> latencies are perfect so it's an e1000 problem.

Or some kind of weird interrupt problem.

Such an interrupt level bug would also account for the TX timeout's
you're seeing btw.

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Tue, 17 Jun 2008 10:32:20 +0200
> 
> > those up to 1000 msec delays can be 'felt' via ssh too, if this 
> > problem triggers then the system is almost unusable via the network. 
> > Local latencies are perfect so it's an e1000 problem.
> 
> Or some kind of weird interrupt problem.
> 
> Such an interrupt level bug would also account for the TX timeout's 
> you're seeing btw.

when i originally reported it i debugged it back to missing e1000 TX 
completion IRQs. I tried various versions of the driver to figure out 
whether new workarounds for e1000 cover it but it was fruitless. There 
is a 1000 msec internal watchdog timer IRQ within e1000 that gets things 
going if it's stuck.

But the line sch_generic.c:222 problem is new. It could be an 
escallation of this same problem - not even the hw-internal watchdog 
timeout fixing up things? So basically two levels of completion failed, 
the third fallback level (a hard reset of the interface) helped things 
get going. High score from me for networking layer robustness :-)

	Ingo

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Tue, 17 Jun 2008 11:27:06 +0200

> when i originally reported it i debugged it back to missing e1000 TX 
> completion IRQs. I tried various versions of the driver to figure out 
> whether new workarounds for e1000 cover it but it was fruitless. There 
> is a 1000 msec internal watchdog timer IRQ within e1000 that gets things 
> going if it's stuck.

Then that explains your latency, the chip is getting stuck and
TX interrupts stop, right.

> But the line sch_generic.c:222 problem is new. It could be an 
> escallation of this same problem - not even the hw-internal watchdog 
> timeout fixing up things? So basically two levels of completion failed, 
> the third fallback level (a hard reset of the interface) helped things 
> get going. High score from me for networking layer robustness :-)

I think it is an escallation of the same problem.  My first thought
is that there must have been some change to the reset logic and it
isn't as foolproof as it used to be, especially under load.

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Tue, 17 Jun 2008 11:27:06 +0200
> 
> > when i originally reported it i debugged it back to missing e1000 TX 
> > completion IRQs. I tried various versions of the driver to figure 
> > out whether new workarounds for e1000 cover it but it was fruitless. 
> > There is a 1000 msec internal watchdog timer IRQ within e1000 that 
> > gets things going if it's stuck.
> 
> Then that explains your latency, the chip is getting stuck and TX 
> interrupts stop, right.

note that the 1000 msecs timer is AFAIK internal to the e1000 
_hardware_, not the driver itself. I.e. probably the firmware detects 
and works around a hung transmitter. This is not detectable from the OS 
(it's not an OS timer), but it can be observed by a lot of testing on a 
totally quiescent system - which i did back then ;-)

i also played a lot with the various knobs of the e1000, none of which 
seemed to help.

/me digs in archives

i reported it to the e1000 folks in 2006:

  Date: Mon, 4 Dec 2006 11:24:00 +0100

against 2.6.19. The original report is below - with a trace and various 
things i tried to debug this.

i eventually got the suggestion from Auke to set RxIntDelay=8 which 
seemed to work around the issue - but since i use a built-in driver i 
dont have that setting here (RxIntDelay=8 is a module load parameter and 
not exposed via Kconfig methods) and the e1000 driver does not seem to 
have changed its default setting for RxIntDelay.

2.6.18-1.2849.fc6 was the last kernel that worked fine.

	Ingo

-------------------->
Date: Wed, 13 Dec 2006 22:09:22 +0100
From: Ingo Molnar <mingo@elte.hu>
To: Auke Kok <auke-jan.h.kok@intel.com>
Subject: Re: e1000: 2.6.19 & long packet latencies
Cc: Jesse Brandeburg <jesse.brandeburg@intel.com>,
        "Ronciak, John" <john.ronciak@intel.com>

Jesse, et al.,

i'm having a weird packet processing latency problem with the e1000 
driver and recent kernels.

The symptom is this: if i connect to a T60 laptop (which has an on-board 
e1000) from the outside, i see large delays in network activity, and ssh 
sessions are very sluggish.

ping latencies show it best under a dynticks kernel (but vanilla 2.6.19 
is affected too):

 titan:~/linux/linux> ping e
 PING europe (10.0.1.15) 56(84) bytes of data.
 64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.340 ms
 64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=757 ms
 64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=1001 ms
 64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=1001 ms
 64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.356 ms
 64 bytes from europe (10.0.1.15): icmp_seq=6 ttl=64 time=2127 ms
 64 bytes from europe (10.0.1.15): icmp_seq=7 ttl=64 time=1002 ms
 64 bytes from europe (10.0.1.15): icmp_seq=8 ttl=64 time=0.320 ms
 64 bytes from europe (10.0.1.15): icmp_seq=9 ttl=64 time=1002 ms
 64 bytes from europe (10.0.1.15): icmp_seq=10 ttl=64 time=2004 ms
 64 bytes from europe (10.0.1.15): icmp_seq=11 ttl=64 time=1002 ms
 64 bytes from europe (10.0.1.15): icmp_seq=12 ttl=64 time=0.303 ms
 64 bytes from europe (10.0.1.15): icmp_seq=13 ttl=64 time=1000 ms
 64 bytes from europe (10.0.1.15): icmp_seq=14 ttl=64 time=2010 ms
 64 bytes from europe (10.0.1.15): icmp_seq=15 ttl=64 time=1009 ms
 64 bytes from europe (10.0.1.15): icmp_seq=16 ttl=64 time=0.283 ms

i have traced this and the 1000/2000 msecs values come from some sort of 
e1000-internal 'heartbeat' interrupt. What seems to happen is that RX 
packet processing is delayed indefinitely and the IRQ just does not 
arrive.

NOTE: the vanilla 2.6.19 kernel shows this too, but the ping delays are 
1/HZ.

here's a (filtered) trace of such a delay. IRQ 0x219 is the e1000 
interrupt:

  <idle>-0     0D.h1 761236us : do_IRQ (c0272a9b 219 0)
 IRQ_219-356   0.... 761412us+: e1000_intr (handle_IRQ_event)
 IRQ_219-356   0.... 761416us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 761418us+: e1000_clean_tx_irq (e1000_intr)
  <idle>-0     0D.h1 2760093us : do_IRQ (c0272a9b 219 0)
 IRQ_219-356   0.... 2760268us+: e1000_intr (handle_IRQ_event)
 IRQ_219-356   0.... 2760273us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 2760275us : e1000_clean_tx_irq (e1000_intr)
  <idle>-0     0D.h1 3804499us : do_IRQ (c0272a9b 219 0)
 IRQ_219-356   0.... 3804674us+: e1000_intr (handle_IRQ_event)
 IRQ_219-356   0.... 3804679us+: e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 3804761us : e1000_clean_tx_irq (e1000_intr)
 IRQ_219-356   0.... 3804763us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 3804765us : e1000_clean_tx_irq (e1000_intr)
softirq--7     0.... 3804810us : net_rx_action (ksoftirqd)
softirq--5     0D.h. 3805425us : do_IRQ (c01598ac 219 0)
 IRQ_219-356   0.... 3805499us+: e1000_intr (handle_IRQ_event)
 IRQ_219-356   0.... 3805504us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 3805506us : e1000_clean_tx_irq (e1000_intr)
 IRQ_219-356   0.... 3805547us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 3805549us : e1000_clean_tx_irq (e1000_intr)
softirq--6     0.... 3805641us : net_tx_action (ksoftirqd)
  <idle>-0     0D.h1 4760910us : do_IRQ (c01451d4 219 0)
 IRQ_219-356   0.... 4761347us+: e1000_intr (handle_IRQ_event)
 IRQ_219-356   0.... 4761352us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 4761353us : e1000_clean_tx_irq (e1000_intr)
  <idle>-0     0D.h1 6761309us : do_IRQ (c0272a9b 219 0)
 IRQ_219-356   0.... 6761483us+: e1000_intr (handle_IRQ_event)
 IRQ_219-356   0.... 6761488us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 6761490us : e1000_clean_tx_irq (e1000_intr)
softirq--5     0D.h. 8760595us : do_IRQ (c0135dc4 219 0)
 IRQ_219-356   0.... 8760676us+: e1000_intr (handle_IRQ_event)
 IRQ_219-356   0.... 8760681us+: e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 8760739us : e1000_clean_tx_irq (e1000_intr)
 IRQ_219-356   0.... 8760740us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 8760742us : e1000_clean_tx_irq (e1000_intr)
softirq--7     0.... 8760885us : net_rx_action (ksoftirqd)
softirq--7     0.... 8760914us+: icmp_rcv (ip_local_deliver)
softirq--7     0.... 8760923us+: icmp_reply (icmp_echo)
  <idle>-0     0D.h1 8761661us : do_IRQ (c0272a9b 219 0)
 IRQ_219-356   0.... 8761833us+: e1000_intr (handle_IRQ_event)
 IRQ_219-356   0.... 8761838us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 8761840us : e1000_clean_tx_irq (e1000_intr)
 IRQ_219-356   0.... 8761875us : e1000_clean_rx_irq (e1000_intr)
 IRQ_219-356   0.... 8761876us : e1000_clean_tx_irq (e1000_intr)
softirq--6     0.... 8761921us : net_tx_action (ksoftirqd)

note that timestamps 2760093us, 4760910us, 6761309us and 8760595us is 
some sort of traffic-independent 'periodic' interrupt that e1000 
generates. That 'housekeeping' interrupt doesnt seem to be doing much. 
The IRQ at 8760595us picks up an icmp packet and replies to it - but the 
icmp packet in reality arrived somewhere between timestamps 6761309us 
and 8760595us - but no IRQ was generated for it!

Suspecting the interrupt-rate controlling bits of the e1000 hw i have 
tried the following tunes too:

 -#define DEFAULT_RDTR                   0
 +#define DEFAULT_RDTR                   1

 -#define DEFAULT_RADV                 128
 +#define DEFAULT_RADV                   1

 -#define DEFAULT_TIDV                  64
 +#define DEFAULT_TIDV                   1

 -#define DEFAULT_TADV                  64
 +#define DEFAULT_TADV                   1

 -#define DEFAULT_ITR                 8000
 +#define DEFAULT_ITR               100000

but they made no difference.

a 2.6.18-ish kernel works fine (2.6.18-1.2849.fc6):

 titan:~/linux/linux> ping e
 PING europe (10.0.1.15) 56(84) bytes of data.
 64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.695 ms
 64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=0.171 ms
 64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=0.184 ms
 64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=0.159 ms
 64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.148 ms

 e1000: 0000:02:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 00:16:41:17:49:d2
 e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection

the precise hardware version is:

 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
        Subsystem: Lenovo ThinkPad T60
        Flags: bus master, fast devsel, latency 0, IRQ 90
        Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 2000 [size=32]
        Capabilities: <access denied>

this laptop has a CoreDuo so i have tried maxcpus=1 too, but it didnt 
make any difference.

Any ideas about what i should try next?

	Ingo

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Kok, Auke]

Ingo Molnar wrote:
> * David Miller <davem@davemloft.net> wrote:
> 
>> From: Ingo Molnar <mingo@elte.hu>
>> Date: Tue, 17 Jun 2008 11:27:06 +0200
>>
>>> when i originally reported it i debugged it back to missing e1000 TX 
>>> completion IRQs. I tried various versions of the driver to figure 
>>> out whether new workarounds for e1000 cover it but it was fruitless. 
>>> There is a 1000 msec internal watchdog timer IRQ within e1000 that 
>>> gets things going if it's stuck.
>> Then that explains your latency, the chip is getting stuck and TX 
>> interrupts stop, right.
> 
> note that the 1000 msecs timer is AFAIK internal to the e1000 
> _hardware_, not the driver itself. I.e. probably the firmware detects 
> and works around a hung transmitter. This is not detectable from the OS 
> (it's not an OS timer), but it can be observed by a lot of testing on a 
> totally quiescent system - which i did back then ;-)
> 
> i also played a lot with the various knobs of the e1000, none of which 
> seemed to help.
> 
> /me digs in archives
> 
> i reported it to the e1000 folks in 2006:
> 
>   Date: Mon, 4 Dec 2006 11:24:00 +0100
> 
> against 2.6.19. The original report is below - with a trace and various 
> things i tried to debug this.
> 
> i eventually got the suggestion from Auke to set RxIntDelay=8 which 
> seemed to work around the issue - but since i use a built-in driver i 
> dont have that setting here (RxIntDelay=8 is a module load parameter and 
> not exposed via Kconfig methods) and the e1000 driver does not seem to 
> have changed its default setting for RxIntDelay.
> 
> 2.6.18-1.2849.fc6 was the last kernel that worked fine.
> 
> 	Ingo
> 
> -------------------->
> Date: Wed, 13 Dec 2006 22:09:22 +0100
> From: Ingo Molnar <mingo@elte.hu>
> To: Auke Kok <auke-jan.h.kok@intel.com>
> Subject: Re: e1000: 2.6.19 & long packet latencies
> Cc: Jesse Brandeburg <jesse.brandeburg@intel.com>,
>         "Ronciak, John" <john.ronciak@intel.com>
> 
> Jesse, et al.,
> 
> i'm having a weird packet processing latency problem with the e1000 
> driver and recent kernels.
> 
> The symptom is this: if i connect to a T60 laptop (which has an on-board 
> e1000) from the outside, i see large delays in network activity, and ssh 
> sessions are very sluggish.
> 
> ping latencies show it best under a dynticks kernel (but vanilla 2.6.19 
> is affected too):
> 
>  titan:~/linux/linux> ping e
>  PING europe (10.0.1.15) 56(84) bytes of data.
>  64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.340 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=757 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=1001 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=1001 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.356 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=6 ttl=64 time=2127 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=7 ttl=64 time=1002 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=8 ttl=64 time=0.320 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=9 ttl=64 time=1002 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=10 ttl=64 time=2004 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=11 ttl=64 time=1002 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=12 ttl=64 time=0.303 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=13 ttl=64 time=1000 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=14 ttl=64 time=2010 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=15 ttl=64 time=1009 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=16 ttl=64 time=0.283 ms
> 
> i have traced this and the 1000/2000 msecs values come from some sort of 
> e1000-internal 'heartbeat' interrupt. What seems to happen is that RX 
> packet processing is delayed indefinitely and the IRQ just does not 
> arrive.
> 
> NOTE: the vanilla 2.6.19 kernel shows this too, but the ping delays are 
> 1/HZ.
> 
> here's a (filtered) trace of such a delay. IRQ 0x219 is the e1000 
> interrupt:
> 
>   <idle>-0     0D.h1 761236us : do_IRQ (c0272a9b 219 0)
>  IRQ_219-356   0.... 761412us+: e1000_intr (handle_IRQ_event)
>  IRQ_219-356   0.... 761416us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 761418us+: e1000_clean_tx_irq (e1000_intr)
>   <idle>-0     0D.h1 2760093us : do_IRQ (c0272a9b 219 0)
>  IRQ_219-356   0.... 2760268us+: e1000_intr (handle_IRQ_event)
>  IRQ_219-356   0.... 2760273us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 2760275us : e1000_clean_tx_irq (e1000_intr)
>   <idle>-0     0D.h1 3804499us : do_IRQ (c0272a9b 219 0)
>  IRQ_219-356   0.... 3804674us+: e1000_intr (handle_IRQ_event)
>  IRQ_219-356   0.... 3804679us+: e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 3804761us : e1000_clean_tx_irq (e1000_intr)
>  IRQ_219-356   0.... 3804763us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 3804765us : e1000_clean_tx_irq (e1000_intr)
> softirq--7     0.... 3804810us : net_rx_action (ksoftirqd)
> softirq--5     0D.h. 3805425us : do_IRQ (c01598ac 219 0)
>  IRQ_219-356   0.... 3805499us+: e1000_intr (handle_IRQ_event)
>  IRQ_219-356   0.... 3805504us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 3805506us : e1000_clean_tx_irq (e1000_intr)
>  IRQ_219-356   0.... 3805547us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 3805549us : e1000_clean_tx_irq (e1000_intr)
> softirq--6     0.... 3805641us : net_tx_action (ksoftirqd)
>   <idle>-0     0D.h1 4760910us : do_IRQ (c01451d4 219 0)
>  IRQ_219-356   0.... 4761347us+: e1000_intr (handle_IRQ_event)
>  IRQ_219-356   0.... 4761352us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 4761353us : e1000_clean_tx_irq (e1000_intr)
>   <idle>-0     0D.h1 6761309us : do_IRQ (c0272a9b 219 0)
>  IRQ_219-356   0.... 6761483us+: e1000_intr (handle_IRQ_event)
>  IRQ_219-356   0.... 6761488us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 6761490us : e1000_clean_tx_irq (e1000_intr)
> softirq--5     0D.h. 8760595us : do_IRQ (c0135dc4 219 0)
>  IRQ_219-356   0.... 8760676us+: e1000_intr (handle_IRQ_event)
>  IRQ_219-356   0.... 8760681us+: e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 8760739us : e1000_clean_tx_irq (e1000_intr)
>  IRQ_219-356   0.... 8760740us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 8760742us : e1000_clean_tx_irq (e1000_intr)
> softirq--7     0.... 8760885us : net_rx_action (ksoftirqd)
> softirq--7     0.... 8760914us+: icmp_rcv (ip_local_deliver)
> softirq--7     0.... 8760923us+: icmp_reply (icmp_echo)
>   <idle>-0     0D.h1 8761661us : do_IRQ (c0272a9b 219 0)
>  IRQ_219-356   0.... 8761833us+: e1000_intr (handle_IRQ_event)
>  IRQ_219-356   0.... 8761838us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 8761840us : e1000_clean_tx_irq (e1000_intr)
>  IRQ_219-356   0.... 8761875us : e1000_clean_rx_irq (e1000_intr)
>  IRQ_219-356   0.... 8761876us : e1000_clean_tx_irq (e1000_intr)
> softirq--6     0.... 8761921us : net_tx_action (ksoftirqd)
> 
> note that timestamps 2760093us, 4760910us, 6761309us and 8760595us is 
> some sort of traffic-independent 'periodic' interrupt that e1000 
> generates. That 'housekeeping' interrupt doesnt seem to be doing much. 
> The IRQ at 8760595us picks up an icmp packet and replies to it - but the 
> icmp packet in reality arrived somewhere between timestamps 6761309us 
> and 8760595us - but no IRQ was generated for it!
> 
> Suspecting the interrupt-rate controlling bits of the e1000 hw i have 
> tried the following tunes too:
> 
>  -#define DEFAULT_RDTR                   0
>  +#define DEFAULT_RDTR                   1
> 
>  -#define DEFAULT_RADV                 128
>  +#define DEFAULT_RADV                   1
> 
>  -#define DEFAULT_TIDV                  64
>  +#define DEFAULT_TIDV                   1
> 
>  -#define DEFAULT_TADV                  64
>  +#define DEFAULT_TADV                   1
> 
>  -#define DEFAULT_ITR                 8000
>  +#define DEFAULT_ITR               100000
> 
> but they made no difference.
> 
> a 2.6.18-ish kernel works fine (2.6.18-1.2849.fc6):
> 
>  titan:~/linux/linux> ping e
>  PING europe (10.0.1.15) 56(84) bytes of data.
>  64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.695 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=0.171 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=0.184 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=0.159 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.148 ms
> 
>  e1000: 0000:02:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 00:16:41:17:49:d2
>  e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
> 
> the precise hardware version is:
> 
>  02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
>         Subsystem: Lenovo ThinkPad T60
>         Flags: bus master, fast devsel, latency 0, IRQ 90
>         Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
>         I/O ports at 2000 [size=32]
>         Capabilities: <access denied>
> 
> this laptop has a CoreDuo so i have tried maxcpus=1 too, but it didnt 
> make any difference.
> 
> Any ideas about what i should try next?
> 

have you tried e1000e?

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* Kok, Auke <auke-jan.h.kok@intel.com> wrote:

> > Any ideas about what i should try next?
> 
> have you tried e1000e?

will try it.

But even it if solves the problem it's a nasty complication: given how 
many times i have to bisect back into the times when there was only 
e1000 around, how do i handle the transition? I have automated bisection 
tools, etc. and i bisect very frequently.

It's a real practical problem for me: if i have E1000E=y in my .config 
and go back to an older kernel, i lose that .config setting in 'make 
oldconfig'. Then when the bisection run happens to go back into the 
E1000E times, 'make oldconfig' picks up E1000E with a default-off 
setting - and things break or work differently.

no other Linux driver i'm using forces me to do that and i rely on many 
of them and i rely on proper 'make oldconfig' behavior on a daily basis. 
Until now i was able to do automatic bisection back for _years_, to the 
v2.6.19 times. You broke that.

And that's just one driver out of thousands of Linux drivers. Imagine 
what happened to bisectability and migration quality if every driver 
version update was this careless about its installed base as 
e1000/e1000e.

The e1000 -> e1000e migration it was not only done in an incompetent, 
amateurish way, you also ignored real feedback and that combined 
together is totally lame and inacceptable behavior in my book. You 
should not expect praise and roses from me as long as you do stupid 
things like that.

	Ingo

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Kok, Auke]

Ingo Molnar wrote:
> * Kok, Auke <auke-jan.h.kok@intel.com> wrote:
> 
>>> Any ideas about what i should try next?
>> have you tried e1000e?
> 
> will try it.
> 
> But even it if solves the problem it's a nasty complication: given how 
> many times i have to bisect back into the times when there was only 
> e1000 around, how do i handle the transition? I have automated bisection 
> tools, etc. and i bisect very frequently.
> 
> It's a real practical problem for me: if i have E1000E=y in my .config 
> and go back to an older kernel, i lose that .config setting in 'make 
> oldconfig'. Then when the bisection run happens to go back into the 
> E1000E times, 'make oldconfig' picks up E1000E with a default-off 
> setting - and things break or work differently.
> 
> no other Linux driver i'm using forces me to do that and i rely on many 
> of them and i rely on proper 'make oldconfig' behavior on a daily basis. 
> Until now i was able to do automatic bisection back for _years_, to the 
> v2.6.19 times. You broke that.
> 
> And that's just one driver out of thousands of Linux drivers. Imagine 
> what happened to bisectability and migration quality if every driver 
> version update was this careless about its installed base as 
> e1000/e1000e.
> 
> The e1000 -> e1000e migration it was not only done in an incompetent, 
> amateurish way, you also ignored real feedback and that combined 
> together is totally lame and inacceptable behavior in my book. You 
> should not expect praise and roses from me as long as you do stupid 
> things like that.

where were you when we discussed this? We took over a year and a half to get to a
final plan and many people responded and provided feedback. In the end Jeff Garzik
and many community members suggested a plan and this is what I implemented. In not
a single way did I force anything down anyones throat. I did exactly what the
community wanted me to do, and in the way that it seemed best by everyone.

You only complain and do not provide a single solution to your problem. Your
continued screaming and whining is totally not productive nor constructive at all,
and frankly is insulting since you completely ignore the fact that we worked with
the the community more than two-year to come to some maintainable situation. All
you do is complain. Direct your problems to the network stack and driver
maintainers since they approved and worked with me to implement the changes.

*** NOTE: I NO LONGER MAINTAIN E1000/E1000E, nor do I represent them or speak for
them. ***

I frankly suggested that you try e1000e because this might provide valuable
information for the people who are taking this ingrateful job after me. This was
meant in a productive and constructive way.

your flame is totally inappropriate and unprofessional. Either come up with a
solution or start working on one, like I did when I took the much hated job as
e1000 maintainer.

I am totally open to suggestions and if needed I will work with the current
e1000/e1000e maintainers on working something out if I see a better solution than
the current situation. Until I see such a thing I can't do much else than ignore
your childish whining.

Auke

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> * Kok, Auke <auke-jan.h.kok@intel.com> wrote:
> 
> > > Any ideas about what i should try next?
> > 
> > have you tried e1000e?
> 
> will try it.

ok, i tried it now, and there's good news: the latency problem seems 
largely fixed by e1000e. (yay!)

with e1000 i got these anomalous latencies:

 64 bytes from europe (10.0.1.15): icmp_seq=10 ttl=64 time=1000 ms
 64 bytes from europe (10.0.1.15): icmp_seq=11 ttl=64 time=0.882 ms
 64 bytes from europe (10.0.1.15): icmp_seq=12 ttl=64 time=1007 ms
 64 bytes from europe (10.0.1.15): icmp_seq=13 ttl=64 time=0.522 ms
 64 bytes from europe (10.0.1.15): icmp_seq=14 ttl=64 time=1003 ms
 64 bytes from europe (10.0.1.15): icmp_seq=15 ttl=64 time=0.381 ms
 64 bytes from europe (10.0.1.15): icmp_seq=16 ttl=64 time=1010 ms

with e1000e i get:

 64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.212 ms
 64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=0.372 ms
 64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=0.815 ms
 64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=0.961 ms
 64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.201 ms
 64 bytes from europe (10.0.1.15): icmp_seq=6 ttl=64 time=0.788 ms

TCP latencies are fine too - ssh feels snappy again.

it still does not have nearly as good latencies as say forcedeth though:

 64 bytes from mercury (10.0.1.13): icmp_seq=1 ttl=64 time=0.076 ms
 64 bytes from mercury (10.0.1.13): icmp_seq=2 ttl=64 time=0.085 ms
 64 bytes from mercury (10.0.1.13): icmp_seq=3 ttl=64 time=0.045 ms
 64 bytes from mercury (10.0.1.13): icmp_seq=4 ttl=64 time=0.053 ms

that's 10 times better packet latencies.

and even an ancient Realtek RTL-8139 over 10 megabit Ethernet (!) has 
better latencies than the e1000e over 1000 megabit:

 64 bytes from pluto (10.0.1.10): icmp_seq=2 ttl=64 time=0.309 ms
 64 bytes from pluto (10.0.1.10): icmp_seq=3 ttl=64 time=0.333 ms
 64 bytes from pluto (10.0.1.10): icmp_seq=4 ttl=64 time=0.329 ms
 64 bytes from pluto (10.0.1.10): icmp_seq=5 ttl=64 time=0.311 ms
 64 bytes from pluto (10.0.1.10): icmp_seq=6 ttl=64 time=0.302 ms

is it done intentionally perhaps? I dont think it makes much sense to 
delay rx/tx processing on a completely idle box for such a long time.

The options i used are:

 CONFIG_E1000=y
 CONFIG_E1000_NAPI=y
 # CONFIG_E1000_DISABLE_PACKET_SPLIT is not set
 CONFIG_E1000E=y
 CONFIG_E1000E_ENABLED=y

> But even it if solves the problem it's a nasty complication: given how 
> many times i have to bisect back into the times when there was only 
> e1000 around, how do i handle the transition? I have automated 
> bisection tools, etc. and i bisect very frequently.

one possibility would be to change 'make oldconfig' to keep old options 
around - as long as they look "unknown" to a particular kernel. It would 
list them in some special "unknown options" section near the end of the 
.config or so. That way the E1000E=y setting could survive a bisection 
run which dives down into older kernel versions. (obviously old kernels 
wont grow this capability magically, so if we do such a change we'll 
have to wait years for it all to trickle through.)

and eventually E1000E could become the default.

	Ingo

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Denys Fedoryshchenko]

> * Ingo Molnar <mingo@elte.hu> wrote:
> with e1000e i get:
> 
>  64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.212 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=0.372 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=0.815 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=0.961 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.201 ms
>  64 bytes from europe (10.0.1.15): icmp_seq=6 ttl=64 time=0.788 ms
> 
> TCP latencies are fine too - ssh feels snappy again.
> 
> it still does not have nearly as good latencies as say forcedeth though:
> 
>  64 bytes from mercury (10.0.1.13): icmp_seq=1 ttl=64 time=0.076 ms
>  64 bytes from mercury (10.0.1.13): icmp_seq=2 ttl=64 time=0.085 ms
>  64 bytes from mercury (10.0.1.13): icmp_seq=3 ttl=64 time=0.045 ms
>  64 bytes from mercury (10.0.1.13): icmp_seq=4 ttl=64 time=0.053 ms
> 
> that's 10 times better packet latencies.
> 
> and even an ancient Realtek RTL-8139 over 10 megabit Ethernet (!) has 
> better latencies than the e1000e over 1000 megabit:
> 
>  64 bytes from pluto (10.0.1.10): icmp_seq=2 ttl=64 time=0.309 ms
>  64 bytes from pluto (10.0.1.10): icmp_seq=3 ttl=64 time=0.333 ms
>  64 bytes from pluto (10.0.1.10): icmp_seq=4 ttl=64 time=0.329 ms
>  64 bytes from pluto (10.0.1.10): icmp_seq=5 ttl=64 time=0.311 ms
>  64 bytes from pluto (10.0.1.10): icmp_seq=6 ttl=64 time=0.302 ms
> 
> is it done intentionally perhaps? I dont think it makes much sense to 
> delay rx/tx processing on a completely idle box for such a long time.
Idle box, ICH8 chipset, e1000e, latest git.

MegaRouterCore-KARAM ~ # ping 192.168.20.26
PING 192.168.20.26 (192.168.20.26) 56(84) bytes of data.
64 bytes from 192.168.20.26: icmp_seq=1 ttl=64 time=0.109 ms
64 bytes from 192.168.20.26: icmp_seq=2 ttl=64 time=0.134 ms
64 bytes from 192.168.20.26: icmp_seq=3 ttl=64 time=0.120 ms
64 bytes from 192.168.20.26: icmp_seq=4 ttl=64 time=0.117 ms
64 bytes from 192.168.20.26: icmp_seq=5 ttl=64 time=0.117 ms
64 bytes from 192.168.20.26: icmp_seq=6 ttl=64 time=0.113 ms

Disabling interrupt moderation
MegaRouterCore-KARAM ~ # ethtool -C eth0 rx-usecs 0
MegaRouterCore-KARAM ~ # ping 192.168.20.26
PING 192.168.20.26 (192.168.20.26) 56(84) bytes of data.
64 bytes from 192.168.20.26: icmp_seq=1 ttl=64 time=0.072 ms
64 bytes from 192.168.20.26: icmp_seq=2 ttl=64 time=0.091 ms
64 bytes from 192.168.20.26: icmp_seq=3 ttl=64 time=0.066 ms
64 bytes from 192.168.20.26: icmp_seq=4 ttl=64 time=0.065 ms
64 bytes from 192.168.20.26: icmp_seq=5 ttl=64 time=0.077 ms
64 bytes from 192.168.20.26: icmp_seq=6 ttl=64 time=0.073 ms

Maybe try the same?
ethtool -C eth0 rx-usecs 0

-- 
------
Technical Manager
Virtual ISP S.A.L.
Lebanon

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* Denys Fedoryshchenko <denys@visp.net.lb> wrote:

> > * Ingo Molnar <mingo@elte.hu> wrote:
> > with e1000e i get:
> > 
> >  64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.212 ms
> >  64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=0.372 ms
> >  64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=0.815 ms
> >  64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=0.961 ms
> >  64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.201 ms
> >  64 bytes from europe (10.0.1.15): icmp_seq=6 ttl=64 time=0.788 ms
> > 
> > TCP latencies are fine too - ssh feels snappy again.
> > 
> > it still does not have nearly as good latencies as say forcedeth though:
> > 
> >  64 bytes from mercury (10.0.1.13): icmp_seq=1 ttl=64 time=0.076 ms
> >  64 bytes from mercury (10.0.1.13): icmp_seq=2 ttl=64 time=0.085 ms
> >  64 bytes from mercury (10.0.1.13): icmp_seq=3 ttl=64 time=0.045 ms
> >  64 bytes from mercury (10.0.1.13): icmp_seq=4 ttl=64 time=0.053 ms
> > 
> > that's 10 times better packet latencies.
> > 
> > and even an ancient Realtek RTL-8139 over 10 megabit Ethernet (!) has 
> > better latencies than the e1000e over 1000 megabit:
> > 
> >  64 bytes from pluto (10.0.1.10): icmp_seq=2 ttl=64 time=0.309 ms
> >  64 bytes from pluto (10.0.1.10): icmp_seq=3 ttl=64 time=0.333 ms
> >  64 bytes from pluto (10.0.1.10): icmp_seq=4 ttl=64 time=0.329 ms
> >  64 bytes from pluto (10.0.1.10): icmp_seq=5 ttl=64 time=0.311 ms
> >  64 bytes from pluto (10.0.1.10): icmp_seq=6 ttl=64 time=0.302 ms
> > 
> > is it done intentionally perhaps? I dont think it makes much sense to 
> > delay rx/tx processing on a completely idle box for such a long time.
> Idle box, ICH8 chipset, e1000e, latest git.
> 
> MegaRouterCore-KARAM ~ # ping 192.168.20.26
> PING 192.168.20.26 (192.168.20.26) 56(84) bytes of data.
> 64 bytes from 192.168.20.26: icmp_seq=1 ttl=64 time=0.109 ms
> 64 bytes from 192.168.20.26: icmp_seq=2 ttl=64 time=0.134 ms
> 64 bytes from 192.168.20.26: icmp_seq=3 ttl=64 time=0.120 ms
> 64 bytes from 192.168.20.26: icmp_seq=4 ttl=64 time=0.117 ms
> 64 bytes from 192.168.20.26: icmp_seq=5 ttl=64 time=0.117 ms
> 64 bytes from 192.168.20.26: icmp_seq=6 ttl=64 time=0.113 ms

ok, that looks much better! i have another box with e1000, ich7:

64 bytes from titan (10.0.1.14): icmp_seq=5 ttl=64 time=0.345 ms
64 bytes from titan (10.0.1.14): icmp_seq=6 ttl=64 time=1.03 ms
64 bytes from titan (10.0.1.14): icmp_seq=7 ttl=64 time=0.383 ms
64 bytes from titan (10.0.1.14): icmp_seq=8 ttl=64 time=0.320 ms
64 bytes from titan (10.0.1.14): icmp_seq=9 ttl=64 time=0.996 ms
64 bytes from titan (10.0.1.14): icmp_seq=10 ttl=64 time=0.248 ms

> Disabling interrupt moderation
> MegaRouterCore-KARAM ~ # ethtool -C eth0 rx-usecs 0
> MegaRouterCore-KARAM ~ # ping 192.168.20.26
> PING 192.168.20.26 (192.168.20.26) 56(84) bytes of data.
> 64 bytes from 192.168.20.26: icmp_seq=1 ttl=64 time=0.072 ms
> 64 bytes from 192.168.20.26: icmp_seq=2 ttl=64 time=0.091 ms
> 64 bytes from 192.168.20.26: icmp_seq=3 ttl=64 time=0.066 ms
> 64 bytes from 192.168.20.26: icmp_seq=4 ttl=64 time=0.065 ms
> 64 bytes from 192.168.20.26: icmp_seq=5 ttl=64 time=0.077 ms
> 64 bytes from 192.168.20.26: icmp_seq=6 ttl=64 time=0.073 ms
> 
> Maybe try the same?
> ethtool -C eth0 rx-usecs 0

well i tend not to tweak my drivers with such options because i want to 
experience and test what 99.9% of our users will experience in the 
field. The reality is that if it's not the default behavior, it's almost 
as if it didnt exist at all.

but even with that tune on e1000e (on the t60, ich7) i still get rather 
large numbers:

earth4:~/s> ping eu
PING europe (10.0.1.15) 56(84) bytes of data.
64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.250 ms
64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=0.250 ms
64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=0.225 ms
64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=0.932 ms
64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.251 ms
64 bytes from europe (10.0.1.15): icmp_seq=6 ttl=64 time=0.915 ms
64 bytes from europe (10.0.1.15): icmp_seq=7 ttl=64 time=0.250 ms
64 bytes from europe (10.0.1.15): icmp_seq=8 ttl=64 time=0.238 ms
64 bytes from europe (10.0.1.15): icmp_seq=9 ttl=64 time=0.390 ms
64 bytes from europe (10.0.1.15): icmp_seq=10 ttl=64 time=0.260 ms

	Ingo

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[David Miller]

From: "Kok, Auke" <auke-jan.h.kok@intel.com>
Date: Wed, 18 Jun 2008 14:25:28 -0700

> You only complain and do not provide a single solution to your
> problem. Your continued screaming and whining is totally not
> productive nor constructive at all, and frankly is insulting since
> you completely ignore the fact that we worked with the the community
> more than two-year to come to some maintainable situation.

I completely and %100 agree with you.

> Until I see such a thing I can't do much else than ignore
> your childish whining.

Join the club.

I'm also ignoring everything he writes until he changes his modus
operandi to one that is more constructive than the pure hurtful
whining he is emitting as of late.

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Denys Fedoryshchenko]

On Thursday 19 June 2008 01:05, Ingo Molnar wrote:
> 
> ok, that looks much better! i have another box with e1000, ich7:
> 
> 64 bytes from titan (10.0.1.14): icmp_seq=5 ttl=64 time=0.345 ms
> 64 bytes from titan (10.0.1.14): icmp_seq=6 ttl=64 time=1.03 ms
> 64 bytes from titan (10.0.1.14): icmp_seq=7 ttl=64 time=0.383 ms
> 64 bytes from titan (10.0.1.14): icmp_seq=8 ttl=64 time=0.320 ms
> 64 bytes from titan (10.0.1.14): icmp_seq=9 ttl=64 time=0.996 ms
> 64 bytes from titan (10.0.1.14): icmp_seq=10 ttl=64 time=0.248 ms
Maybe there is some flow-control involved?
ethtool -S eth0 ?

This is Interrupt throttling i guess in e1000. In e1000 also parameters, but available only on insmod stage
parm:           TxIntDelay:Transmit Interrupt Delay (array of int)
parm:           TxAbsIntDelay:Transmit Absolute Interrupt Delay (array of int)
parm:           RxIntDelay:Receive Interrupt Delay (array of int)
parm:           RxAbsIntDelay:Receive Absolute Interrupt Delay (array of int)
parm:           InterruptThrottleRate:Interrupt Throttling Rate (array of int)


> well i tend not to tweak my drivers with such options because i want to 
> experience and test what 99.9% of our users will experience in the 
> field. The reality is that if it's not the default behavior, it's almost 
> as if it didnt exist at all.

Each coin have two sides. On one side - low latencies(difference 1ms, it is matter anywhere?)
, on another - performance.


> 
> but even with that tune on e1000e (on the t60, ich7) i still get rather 
> large numbers:
> 
> earth4:~/s> ping eu
> PING europe (10.0.1.15) 56(84) bytes of data.
> 64 bytes from europe (10.0.1.15): icmp_seq=1 ttl=64 time=0.250 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=2 ttl=64 time=0.250 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=3 ttl=64 time=0.225 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=4 ttl=64 time=0.932 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=5 ttl=64 time=0.251 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=6 ttl=64 time=0.915 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=7 ttl=64 time=0.250 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=8 ttl=64 time=0.238 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=9 ttl=64 time=0.390 ms
> 64 bytes from europe (10.0.1.15): icmp_seq=10 ttl=64 time=0.260 ms


Is all this hosts on same switch? Is the switch manageable or not?
For example i am having problems with packetloss on long fiber link between two cheap Linksys switches. 
Without flow-control  i cannot survive, and as result i have 1-2ms additional delay on load, and +-0.500ms jitter "inside" this switches (probably from switches).

There is many things matter. Maybe even processor sleep latencies involved? bus latency, PCI latency, whatever. 

Also on laptops is dynamic frequency running (Speedstep)
with 600 Mhz PentiumM (Speedstep - ondemand)
64 bytes from 127.0.0.1: icmp_seq=17 ttl=64 time=0.017 ms
full speed 1.7 Ghz
64 bytes from 127.0.0.1: icmp_seq=33 ttl=64 time=0.007 ms

on network also i see difference -0.030ms when i am running burnP6 (from CPUburn package).





-- 
------
Technical Manager
Virtual ISP S.A.L.
Lebanon

Re: [E1000-devel] [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Ingo Molnar]

* Kok, Auke <auke-jan.h.kok@intel.com> wrote:

> You only complain and do not provide a single solution to your 
> problem. [...]

i have reported the problem and even provided a fix.

I have triggered an e1000/e1000e related problem that got introduced in 
the v2.6.25 merge window - one of my testboxes came up with no 
networking and it took me an hour to figure out why. (i wasnt 
particularly focusing on e1000, i just happened to hit that bug in 9 
million lines of Linux kernel code)

I have reported it here, two and a half months ago:

    http://lkml.org/lkml/2008/4/8/256

I even showed you which commit introduced the problem and gave you a 
oneliner fix that i tested (it solved the problem):

    http://bugzilla.kernel.org/attachment.cgi?id=15704&amp;action=view

You were Cc:-ed to that. (attached below again for reference) The bug 
was added to the regression list of v2.6.25. I never expected to spend 
more than 10 minutes on this problem once i found out what's happening - 
we fix dozens of bugs like this per stable kernel release.

I just checked latest -git, my fix is still not upstream (or any 
equivalent solution - i really dont mind how it's solved and i'm not 
maintaining this code).

no alternative patch was sent to me - i offered to test any solution 
back then.

FYI, since i first reported it i've been hit by that problem roughly a 
dozen times. (it happened sporadically so i forgot about it - until i 
again had a system come up with no networking.) It caused me lost time 
and lost work that could have been spent on better things.

	Ingo

------------------------>
Subject: e1000=y && e1000e=m regression fix
From: Ingo Molnar <mingo@elte.hu>
Date: Wed Apr 09 21:09:35 CEST 2008

fix a regression from v2.6.24: do not transfer the e1000e PCI IDs from 
e1000 to e1000e if e1000 is built-in and e1000e is a module.

Built-in drivers take precedence over modules in many ways - and in this 
case it's clear that the user intended the e1000 driver to be the 
primary one. "Silently change behavior and break existing configs" is 
never a good migration strategy. Most users will use distro kernels that 
are not affected by this problem at all - nor are they affected by this 
patch - but this problem can hit users and developers who build their 
kernels themselves and migrate from v2.6.24 to v2.6.25.

this fixes: http://bugzilla.kernel.org/show_bug.cgi?id=10427

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 drivers/net/Kconfig |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux-x86.q/drivers/net/Kconfig
===================================================================
--- linux-x86.q.orig/drivers/net/Kconfig
+++ linux-x86.q/drivers/net/Kconfig
@@ -2022,7 +2022,7 @@ config E1000E
 	  will be called e1000e.
 
 config E1000E_ENABLED
-	def_bool E1000E != n
+	def_bool E1000E = y || ((E1000E != n) && (E1000 = E1000E))
 
 config IP1000
 	tristate "IP1000 Gigabit Ethernet support"

Re: [TCP]: TCP_DEFER_ACCEPT causes leak sockets

[Jarek Poplawski]

On 19-06-2008 00:12, David Miller wrote:
> From: "Kok, Auke" <auke-jan.h.kok@intel.com>
> Date: Wed, 18 Jun 2008 14:25:28 -0700
> 
>> You only complain and do not provide a single solution to your
>> problem.

Technically, Ingo has asked for a solution (but btw. he gave some
proposal), so this argument is wrong.

>> Your continued screaming and whining is totally not
>> productive nor constructive at all,

Actually, screaming and whining often "helps" to have things done,
so this argument is wrong too.

>> and frankly is insulting

This one is right.

>> since
>> you completely ignore the fact that we worked with the the community
>> more than two-year to come to some maintainable situation.

Two-year work doesn't guarantee the solution is right (but it might be
right),so - wrong argument.

> 
> I completely and %100 agree with you.

I give 25...

Regards,
Jarek P.