netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Ingo Molnar <mingo@elte.hu>
To: David Miller <davem@davemloft.net>
Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370
Date: Mon, 21 Jul 2008 15:45:06 +0200	[thread overview]
Message-ID: <20080721134506.GA27598@elte.hu> (raw)
In-Reply-To: <20080721133059.GA30637@elte.hu>


* Ingo Molnar <mingo@elte.hu> wrote:

> David,
> 
> -tip testing on latest -git (v2.6.26-5253-g14b395e) triggered the 
> following boot crash on a Core2Duo 64-bit testsystem:
> 
> ADDRCONF(NETDEV_UP): eth0: link is not ready
> eth0: Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> ------------[ cut here ]------------
> Kernel BUG at ffffffff8079afb1 [verbose debug info unavailable]
> invalid opcode: 0000 [1] SMP 
> CPU 0 
> Pid: 7, comm: events/0 Not tainted 2.6.26-rc8 #21302
> RIP: 0010:[<ffffffff8079afb1>]  [<ffffffff8079afb1>] __netif_schedule+0xd/0x64

note, my tests have also triggered another boot crash on the same 
system, using the same config:

PM: Removing info for No Bus:phy0
mac80211_hwsim: ieee80211_register_hw failed (-2)
BUG: unable to handle kernel NULL pointer dereference at 0000000000000370
IP: [<ffffffff808da9f1>] rollback_registered+0x2a/0xd6
PGD 0 
Oops: 0000 [1] SMP 
CPU 1 
Pid: 1, comm: swapper Not tainted 2.6.26-tip-00013-g6de15c6-dirty #21290
RIP: 0010:[<ffffffff808da9f1>]  [<ffffffff808da9f1>] rollback_registered+0x2a/0xd6
RSP: 0018:ffff88003f83fe00  EFLAGS: 00010212
RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffff88003d4baed8
RDX: ffffffff80979f1d RSI: 0000000000000046 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff80d6f4a0 R09: ffff880004576800
R10: 0000000000000000 R11: ffffffff80406afe R12: 0000000000000000
R13: ffff88003d4bb9a0 R14: 0000000000000000 R15: 0000000000000008
FS:  0000000000000000(0000) GS:ffff88003f829160(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000370 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo ffff88003f83e000, task ffff88003f824000)
Stack:  0000000000000000 ffffffff808daacf ffff88003d4ba2c0 ffffffff8097e1da
 ffff88003d4bb9a0 ffffffff8060eb76 00000000fffffffe ffff88003d4ba2c0
 ffff88003d4bb9e0 ffffffff811be87a ffff88003f83fea0 ffffffff8024e672
Call Trace:
 [<ffffffff808daacf>] unregister_netdevice+0x32/0x77
 [<ffffffff8097e1da>] ieee80211_unregister_hw+0x35/0xd4
 [<ffffffff8060eb76>] mac80211_hwsim_free+0x1d/0x6a
 [<ffffffff811be87a>] init_mac80211_hwsim+0x2df/0x2f0
 [<ffffffff8024e672>] getnstimeofday+0x38/0x95
 [<ffffffff8024c76a>] ktime_get_ts+0x21/0x49
 [<ffffffff811be59b>] init_mac80211_hwsim+0x0/0x2f0
 [<ffffffff8020a042>] do_one_initcall+0x42/0x13b
 [<ffffffff80247105>] __queue_work+0x23/0x33
 [<ffffffff811a09e0>] kernel_init+0x203/0x271
 [<ffffffff80234e73>] schedule_tail+0x28/0x60
 [<ffffffff80211079>] child_rip+0xa/0x11
 [<ffffffff811a07dd>] kernel_init+0x0/0x271
 [<ffffffff8021106f>] child_rip+0x0/0x11

Code: c3 53 48 89 fb e8 38 78 00 00 85 c0 75 1d ba ce 0e 00 00 48 c7 c6 b5 e9 d4 80 48 c7 c7 4f 85 ca 80 e8 f3 f1 95 ff e8 aa 7c 93 ff <83> bb 70 03 00 00 00 75 15 48 89 da 48 89 de 48 c7 c7 35 eb d4 
RIP  [<ffffffff808da9f1>] rollback_registered+0x2a/0xd6
 RSP <ffff88003f83fe00>
CR2: 0000000000000370
Kernel panic - not syncing: Fatal exception
Rebooting in 1 seconds..Press any key to enter the menu

this crash led to the bisection result i posted in the previous mail. 
This could be a dual bug and one of the crashes masks the other one.

Maybe the __netif_schedule bug is already fixed and when i tried to 
bisect the rollback_registered crash i ran into it as bisection went 
back into networking history?

Same config as before:

  http://redhat.com/~mingo/misc/config-Mon_Jul_21_13_59_54_CEST_2008.bad

Full crashlog:

  http://redhat.com/~mingo/misc/crash-Mon_Jul_21_13_59_52_CEST_2008.log

If the __netif_schedule() bug is already fixed by a later commit then i 
could attempt to bisect this other crash as well, given an sha1 that i 
could cherry-pick into each bisection point.

Thanks,

	Ingo

  reply	other threads:[~2008-07-21 13:45 UTC|newest]

Thread overview: 204+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-07-20 17:44 [GIT]: Networking David Miller
2008-07-20 17:59 ` Arjan van de Ven
2008-07-20 23:52   ` David Miller
2008-07-21 20:32   ` David Miller
2008-07-21  0:54 ` Linus Torvalds
2008-07-21  1:03   ` David Miller
2008-07-21  1:09     ` Alexey Dobriyan
2008-07-21  1:14       ` David Miller
2008-07-21  1:22         ` Alexey Dobriyan
2008-07-21  2:40         ` Alexey Dobriyan
2008-07-21  2:48           ` David Miller
2008-07-21  5:11             ` David Miller
2008-07-21  9:48               ` Alexander Beregalov
2008-07-21 10:16                 ` Ben Hutchings
2008-07-21 15:35                   ` David Miller
2008-07-21 16:04                     ` Alexander Beregalov
2008-07-21 11:57               ` Alexey Dobriyan
2008-07-21 15:27                 ` David Miller
2008-07-21 16:49               ` Linus Torvalds
2008-07-21 16:53                 ` David Miller
2008-07-21  1:20     ` Patrick McHardy
2008-07-21 11:28       ` Stefan Richter
2008-07-21 11:45       ` James Morris
2008-07-21 12:05         ` Patrick McHardy
2008-07-21 17:28           ` David Miller
2008-07-21 17:40             ` Linus Torvalds
2008-07-21 20:33               ` Patrick McHardy
2008-07-23 23:42                 ` David Miller
2008-07-21  1:07   ` Linus Torvalds
2008-07-21  1:17     ` David Miller
2008-07-21  8:36 ` iwlwifi: fix build bug in "iwlwifi: fix LED stall" Ingo Molnar
2008-07-21 10:02   ` Winkler, Tomas
2008-07-21 10:53     ` Ingo Molnar
2008-07-21 12:12   ` [PATCH] iwlwifi: RS small compile warnings without CONFIG_IWLWIFI_DEBUG Tomas Winkler
2008-07-21 12:12     ` [PATCH] iwlwifi: " Tomas Winkler
2008-07-21 12:12       ` [PATCH] iwlwifi: compilation error when CONFIG_IWLWIFI_DEBUG is not set Tomas Winkler
2008-07-21 13:30 ` [crash, bisected] Kernel BUG at ffffffff8079afb1 (__netif_schedule()) Ingo Molnar
2008-07-21 13:45   ` Ingo Molnar [this message]
2008-07-21 14:30     ` [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370 Ingo Molnar
2008-07-21 15:04       ` Ingo Molnar
2008-07-21 15:24         ` David Miller
2008-07-21 18:18           ` Ian Schram
2008-07-21 19:06             ` Ingo Molnar
     [not found]               ` <20080721190646.GA19044-X9Un+BFzKDI@public.gmane.org>
2008-07-21 19:13                 ` Larry Finger
     [not found]                   ` <4884DFEE.4060003-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-21 19:34                     ` Ingo Molnar
     [not found]                       ` <20080721193425.GA29287-X9Un+BFzKDI@public.gmane.org>
2008-07-21 19:43                         ` Larry Finger
     [not found]                           ` <4884E6E6.7010800-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-21 19:47                             ` Linus Torvalds
2008-07-21 20:15                               ` David Miller
     [not found]                               ` <alpine.LFD.1.10.0807211246280.31863-5CScLwifNT1QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org>
2008-07-21 20:28                                 ` Larry Finger
2008-07-21 20:21                             ` David Miller
     [not found]                               ` <20080721.132100.128525904.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-21 20:38                                 ` Larry Finger
2008-07-21 20:46                                   ` David Miller
2008-07-21 20:51                                     ` Patrick McHardy
2008-07-21 21:01                                       ` David Miller
2008-07-21 21:06                                         ` Patrick McHardy
     [not found]                                           ` <4884FA3F.70109-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-21 21:35                                             ` Patrick McHardy
     [not found]                                               ` <48850121.9090106-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-21 21:42                                                 ` Patrick McHardy
2008-07-21 21:51                                                 ` Larry Finger
     [not found]                                                   ` <488504D9.5090100-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-21 22:04                                                     ` Patrick McHardy
     [not found]                                                       ` <488507FB.9050101-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-21 22:40                                                         ` Larry Finger
     [not found]                                                           ` <4885104A.2070201-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-21 23:15                                                             ` David Miller
2008-07-22  6:34                                                               ` Larry Finger
     [not found]                                                                 ` <48857F74.2040406-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-22 10:51                                                                   ` Jarek Poplawski
2008-07-22 11:32                                                                 ` David Miller
2008-07-22 12:52                                                                   ` Larry Finger
2008-07-22 20:43                                                                     ` David Miller
2008-07-22 13:02                                                                   ` Larry Finger
     [not found]                                                                     ` <4885DA49.50703-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-22 14:53                                                                       ` Patrick McHardy
     [not found]                                                                         ` <4885F46A.30309-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-22 21:17                                                                           ` David Miller
2008-07-22 16:39                                                                       ` Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98() Larry Finger
     [not found]                                                                         ` <48860D4B.8070003-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-22 17:20                                                                           ` Patrick McHardy
2008-07-22 18:39                                                                             ` Larry Finger
2008-07-22 18:44                                                                               ` Patrick McHardy
     [not found]                                                                                 ` <48862A76.8030307-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-22 19:30                                                                                   ` Larry Finger
2008-07-22 23:04                                                                               ` David Miller
2008-07-23  6:20                                                                                 ` Jarek Poplawski
2008-07-23  7:59                                                                                   ` David Miller
     [not found]                                                                                     ` <20080723.005921.113868915.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-23  8:54                                                                                       ` Jarek Poplawski
     [not found]                                                                                         ` <20080723085452.GB4561-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-07-23  9:03                                                                                           ` Peter Zijlstra
2008-07-23  9:35                                                                                             ` Jarek Poplawski
     [not found]                                                                                               ` <20080723093459.GC4561-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-07-23  9:50                                                                                                 ` Peter Zijlstra
2008-07-23 10:13                                                                                                   ` Jarek Poplawski
2008-07-23 10:58                                                                                                     ` Peter Zijlstra
2008-07-23 11:35                                                                                                       ` Jarek Poplawski
     [not found]                                                                                                         ` <20080723113519.GE4561-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-07-23 11:49                                                                                                           ` Jarek Poplawski
2008-07-23 20:16                                                                                                             ` David Miller
2008-07-23 20:43                                                                                                               ` Jarek Poplawski
     [not found]                                                                                                                 ` <20080723204335.GA14788-dUp/P3zyUuaNj9Bq2fkWzw@public.gmane.org>
2008-07-23 20:55                                                                                                                   ` David Miller
     [not found]                                                                                                               ` <20080723.131607.79681752.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-24  9:10                                                                                                                 ` Peter Zijlstra
2008-07-24  9:20                                                                                                                   ` David Miller
     [not found]                                                                                                                     ` <20080724.022040.23129457.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-24  9:27                                                                                                                       ` Peter Zijlstra
2008-07-24  9:32                                                                                                                         ` David Miller
     [not found]                                                                                                                           ` <20080724.023210.229338550.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-24 10:08                                                                                                                             ` Peter Zijlstra
2008-07-24 10:38                                                                                                                               ` Nick Piggin
     [not found]                                                                                                                                 ` <200807242038.36693.nickpiggin-/E1597aS9LT0CCvOHzKKcA@public.gmane.org>
2008-07-24 10:55                                                                                                                                   ` Miklos Szeredi
     [not found]                                                                                                                                     ` <E1KLyUI-0006ZS-Cg-8f8m9JG5TPIdUIPVzhDTVZP2KDSNp7ea@public.gmane.org>
2008-07-24 11:06                                                                                                                                       ` Nick Piggin
     [not found]                                                                                                                                         ` <200807242106.52672.nickpiggin-/E1597aS9LT0CCvOHzKKcA@public.gmane.org>
2008-08-01 21:10                                                                                                                                           ` Paul E. McKenney
2008-07-24 10:59                                                                                                                                   ` Peter Zijlstra
2008-08-01 21:10                                                                                                                                 ` Paul E. McKenney
2008-07-23 20:14                                                                                                       ` David Miller
     [not found]                                                                                                         ` <20080723.131441.200166513.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-24  7:00                                                                                                           ` Peter Zijlstra
2008-07-25 17:04                                                                                                         ` Ingo Oeser
     [not found]                                                                                                           ` <200807251904.37302.netdev-BkyiQQGWkgE@public.gmane.org>
2008-07-25 18:36                                                                                                             ` Jarek Poplawski
2008-07-25 19:16                                                                                                               ` Johannes Berg
     [not found]                                                                                                                 ` <1217013384.4758.5.camel-YfaajirXv214zXjbi5bjpg@public.gmane.org>
2008-07-25 19:34                                                                                                                   ` Jarek Poplawski
2008-07-25 19:36                                                                                                                     ` Johannes Berg
2008-07-25 20:01                                                                                                                       ` Jarek Poplawski
     [not found]                                                                                                                         ` <20080725200137.GC3107-dUp/P3zyUuaNj9Bq2fkWzw@public.gmane.org>
2008-07-26  9:18                                                                                                                           ` David Miller
2008-07-26 10:53                                                                                                                             ` Jarek Poplawski
     [not found]                                                                                                                             ` <20080726.021846.236624483.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-26 13:18                                                                                                                               ` Jarek Poplawski
     [not found]                                                                                                                                 ` <20080726131838.GA2873-dUp/P3zyUuaNj9Bq2fkWzw@public.gmane.org>
2008-07-27  0:34                                                                                                                                   ` David Miller
     [not found]                                                                                                                                     ` <20080726.173434.48036095.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-27 20:37                                                                                                                                       ` Jarek Poplawski
     [not found]                                                                                                                                         ` <20080727203757.GA2527-dUp/P3zyUuaNj9Bq2fkWzw@public.gmane.org>
2008-07-31 12:29                                                                                                                                           ` David Miller
     [not found]                                                                                                                                             ` <20080731.052932.110299354.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-31 12:38                                                                                                                                               ` Nick Piggin
2008-07-31 12:44                                                                                                                                                 ` David Miller
2008-08-01  4:27                                                                                                                                             ` David Miller
     [not found]                                                                                                                                               ` <20080731.212729.138736823.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-08-01  7:09                                                                                                                                                 ` Peter Zijlstra
2008-08-01  6:48                                                                                                                                             ` Jarek Poplawski
     [not found]                                                                                                                                               ` <20080801064810.GA4435-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-08-01  7:00                                                                                                                                                 ` David Miller
2008-08-01  7:01                                                                                                                                               ` Jarek Poplawski
     [not found]                                                                                                                                                 ` <20080801070150.GB4435-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-08-01  7:01                                                                                                                                                   ` David Miller
2008-08-01  7:41                                                                                                                                                     ` Jarek Poplawski
2008-07-25  6:20                                     ` [lockdep warning] AOE / networking: aoenet_xmit: noop_qdisc.q.lock, INFO: inconsistent lock state at 0000000000000370 Ingo Molnar
2008-07-25  6:25                                       ` David Miller
2008-07-25  7:26                                         ` Ingo Molnar
2008-07-25  8:23                                           ` David Miller
     [not found]       ` <20080721143023.GA32451-X9Un+BFzKDI@public.gmane.org>
2008-07-21 15:10         ` [crash] BUG: unable to handle kernel NULL pointer dereference " David Miller
2008-07-21 18:23     ` [crash] kernel BUG at net/core/dev.c:1328! Ingo Molnar
2008-07-21 18:35       ` Linus Torvalds
2008-07-21 18:46         ` Ingo Molnar
2008-07-21 19:30           ` Ingo Molnar
2008-07-22 11:21           ` [TCP bug] stuck distcc connections in latest -git Ingo Molnar
2008-07-22 13:45             ` David Newall
2008-07-22 13:57               ` Ingo Molnar
2008-07-22 14:54                 ` David Newall
2008-07-22 15:34                   ` Ingo Molnar
2008-07-22 21:12                     ` Willy Tarreau
2008-07-23  8:26                       ` Ingo Molnar
2008-07-24  6:04                         ` [TCP bug, regression] " Ingo Molnar
2008-07-24  6:32                           ` Ingo Molnar
2008-07-24  7:33                             ` Willy Tarreau
2008-07-24  8:35                               ` Ingo Molnar
2008-07-24  7:53                             ` Herbert Xu
2008-07-24  8:24                               ` Willy Tarreau
2008-07-24  8:27                               ` Ingo Molnar
2008-07-24  8:36                                 ` David Miller
2008-07-24  9:05                           ` Herbert Xu
2008-07-24  9:22                             ` David Miller
2008-07-24  9:34                               ` Ingo Molnar
2008-07-24 11:56                                 ` [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference Ingo Molnar
2008-07-24 11:59                                   ` Ingo Molnar
2008-07-24 12:03                                     ` Patrick McHardy
2008-07-24 12:22                                       ` Herbert Xu
2008-07-24 12:40                                         ` Pekka Enberg
2008-07-24 12:50                                           ` Herbert Xu
2008-07-24 12:56                                             ` Nick Piggin
2008-07-24 13:04                                               ` Herbert Xu
2008-07-24 13:13                                                 ` Nick Piggin
2008-07-24 13:32                                                   ` Pekka Enberg
2008-07-24 19:21                                                     ` Matt Mackall
2008-07-25  9:09                                                       ` Nick Piggin
2008-07-24 13:11                                             ` Matt Mackall
2008-07-24 14:37                                               ` Herbert Xu
2008-07-24 17:47                                                 ` Matt Mackall
2008-07-25  1:39                                                   ` Herbert Xu
2008-07-25  2:59                                                     ` Matt Mackall
2008-07-24 12:44                                       ` Pekka Enberg
2008-07-24 12:49                                         ` Patrick McHardy
2008-07-24 13:23                                           ` Pekka Enberg
2008-07-24 13:31                                             ` Patrick McHardy
2008-07-24 13:34                                               ` Pekka Enberg
2008-07-24 18:51                                                 ` Andrew Morton
2008-07-24 18:55                                                   ` Pekka Enberg
2008-07-24 20:58                                                     ` David Miller
2008-07-25  8:02                                                     ` Dieter Ries
2008-07-25 10:41                                                       ` Pekka Enberg
2008-07-24 19:35                                                   ` Ingo Molnar
2008-07-26 16:09                                                     ` Patrick McHardy
2008-07-26 17:34                                                       ` Ingo Molnar
2008-07-26 13:43                                                   ` Patrick McHardy
2008-07-24 21:13                                           ` Linus Torvalds
2008-07-24 22:09                                             ` David Miller
2008-07-26 13:47                                             ` Patrick McHardy
2008-08-01 21:10                                             ` Paul E. McKenney
2008-07-24 14:23                                     ` Ingo Molnar
2008-07-24 15:23                                       ` Patrick McHardy
2008-07-24 15:32                                         ` Ingo Molnar
2008-07-24 15:34                                           ` Patrick McHardy
2008-07-24 18:00                                           ` Krzysztof Oledzki
2008-07-24 13:01                               ` [TCP bug, regression] stuck distcc connections in latest -git Willy Tarreau
2008-07-24  9:25                             ` Ingo Molnar
2008-07-24  9:29                               ` David Miller
2008-07-24 11:12                               ` Herbert Xu
2008-07-24  9:36                             ` Ilpo Järvinen
2008-07-24 10:03                               ` Ilpo Järvinen
2008-07-21 19:00         ` [crash] kernel BUG at net/core/dev.c:1328! David Miller
2008-07-21 19:20           ` Stefan Richter
2008-07-21 20:11             ` David Miller
2008-07-21 21:26               ` Stefan Richter
2008-07-21 19:44           ` Ingo Molnar
2008-07-21 20:20             ` David Miller
2008-07-21 15:07   ` [crash, bisected] Kernel BUG at ffffffff8079afb1 (__netif_schedule()) David Miller
2008-07-21 13:50 ` [GIT]: Networking Ingo Molnar
2008-07-21 14:15   ` Stefan Richter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20080721134506.GA27598@elte.hu \
    --to=mingo@elte.hu \
    --cc=akpm@linux-foundation.org \
    --cc=davem@davemloft.net \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).