* [BUG] null pointer dereference in tcp_gso_segment()
@ 2014-01-22 21:46 Arnaud Ebalard
2014-01-22 21:57 ` Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Arnaud Ebalard @ 2014-01-22 21:46 UTC (permalink / raw)
To: David Miller, Eric Dumazet, Daniel Borkmann, Herbert Xu, Willy Tarreau
Cc: netdev
Hi,
The following is a backtrace I got while doing a simple transfer from my
NAS (ARMv7-based ReadyNAS 102) running a 3.13.0 kernel. Usage was a
simple file served by nginx. I managed to get a second identical one
in another session. As this did not looked like some random race
condition or some missing lock, I decided to take a look. Details are
given below. FWIW, driver is mvneta (w/ fixes and performance patches
you accepted for v3.14, David) and ethtool reports gso is enabled and tso
disabled.
[ 943.860383] Unable to handle kernel NULL pointer dereference at virtual address 00000090
[ 943.868497] pgd = c0004000
[ 943.871217] [00000090] *pgd=00000000
[ 943.874813] Internal error: Oops: 17 [#1] ARM
[ 943.879175] Modules linked in:
[ 943.882248] CPU: 0 PID: 0 Comm: swapper Not tainted 3.13.0.rn102-00594-ga0fa1dd3cdbc-dirty #63
[ 943.890873] task: c0775620 ti: c0768000 task.ti: c0768000
[ 943.896291] PC is at tcp_gso_segment+0x1d8/0x390
[ 943.900925] LR is at skb_segment+0x510/0x788
[ 943.905202] pc : [<c04b2094>] lr : [<c0448b90>] psr: 200f0113
[ 943.905202] sp : c0769aa8 ip : 00005b27 fp : 00000001
[ 943.916697] r10: 00005b27 r9 : 00000868 r8 : dfab5cb0
[ 943.921930] r7 : 00000000 r6 : 4ef79279 r5 : 00000020 r4 : deeba830
[ 943.928467] r3 : 000005a8 r2 : df993200 r1 : c04a63f8 r0 : 000005a8
[ 943.935005] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 943.942325] Control: 10c5387d Table: 1534c019 DAC: 00000015
[ 943.948079] Process swapper (pid: 0, stack limit = 0xc0768238)
[ 943.953921] Stack: (0xc0769aa8 to 0xc076a000)
[ 943.958288] 9aa0: c077000c 000005a8 0000e397 d52f3540 c0771230 00001c48
[ 943.966481] 9ac0: 00000006 deeba830 0000c6bc 00000014 0000009c 00000000 00000014 c04beeb4
[ 943.974674] 9ae0: c0fbb8ec dec2ab80 0000008e 00000000 00000001 00004803 00000002 deeba830
[ 943.982867] 9b00: c07739cc deeba830 df27d15c df1fd800 c05a045c c0452dcc 00000004 c07739e0
[ 943.991061] 9b20: 00004803 00004803 00000002 00010000 00000000 deeba830 df27d15c c04530e8
[ 943.999254] 9b40: 00000001 c046fc40 c0492410 df1fd800 00000000 00000000 00000000 def06e00
[ 944.007447] 9b60: df1fd800 deeba830 df1fd800 df27d15c 00000010 c0469af4 def06e00 00000000
[ 944.015640] 9b80: 00000042 00000000 df27d15c df1fd800 deeba830 c04535b8 def06e60 000333c5
[ 944.023834] 9ba0: 00000000 dfaf5180 dfaf51fc 0000000e 00000000 deeba830 df1fd800 def35580
[ 944.032027] 9bc0: 00000010 c0492604 80000000 0b50a8c0 00000000 deeba830 df993200 00000000
[ 944.040221] 9be0: df993404 def35580 00000000 009c0000 00000000 c0492a2c deeba830 c0492b5c
[ 944.048414] 9c00: deeba780 df993200 deeba780 00000020 00000000 0000fb7b 00000020 df993200
[ 944.056608] 9c20: deeba830 c0778368 00000000 def35580 00000000 c04a76e8 c3a0f5c7 134b291d
[ 944.064802] 9c40: 00000001 00000000 00000002 00000000 00000000 0000fb7b 002411ee 00000000
[ 944.072995] 9c60: 79e31870 df993200 deeba780 000005a8 4ef79279 00001c48 00000000 000021f0
[ 944.081188] 9c80: 00000000 c04a7d68 c39c239b 134b291d 00000006 000016a0 000021f0 00000001
[ 944.089381] 9ca0: 00000000 df9932bc 00000002 00000000 00000006 134b291d d52f3540 df993200
[ 944.097575] 9cc0: d567e7e4 d52f3540 00000020 d567e7d0 d52f3540 0000ddc7 c0770618 c04a86d8
[ 944.105768] 9ce0: 00000020 df993200 df993200 c04a4954 00000000 00000020 00000001 00000002
[ 944.113961] 9d00: d52f3540 df993200 def35a00 df993200 d567e7d0 d52f3540 0000ddc7 c04ac61c
[ 944.122154] 9d20: c048da1c c046fc40 c048da1c c046fc40 dec12140 c0770b64 d52f3540 d52f3540
[ 944.130348] 9d40: c07a3dc0 00000000 df993200 c04aede0 df1fd800 c046fcc0 00000000 c0769d74
[ 944.138541] 9d60: c048da1c 80000000 c048d734 c0770b64 0000000e c0770b64 3050a8c0 c05aacb4
[ 944.146734] 9d80: c0773470 00000000 c07a3dc0 d52f3540 d52f3540 00000000 c0770618 c048dab0
[ 944.154927] 9da0: d567e7d0 c07739e8 c0770628 df1fd800 d52f3540 c048d85c df1b4400 c0770614
[ 944.163120] 9dc0: c0770614 c07739e8 c0770628 df1fd800 00000008 c044f018 00000003 00000000
[ 944.171314] 9de0: e1424800 42000000 df1fd800 d52f3540 00000001 c0770628 00000000 d52f3540
[ 944.179507] 9e00: 00000003 df3bd400 e1424800 42000000 df1fd800 d52f3540 00000001 c045153c
[ 944.187700] 9e20: 00000004 c0019430 e1424000 c0382238 00000000 00000000 00000002 dfbc4600
[ 944.195893] 9e40: 00000001 df1fdbe0 00000001 00000042 00000000 00000000 dee34000 00000040
[ 944.204086] 9e60: df1fdbe0 00000100 00000000 00000100 df3bd400 c079b684 00000000 c0382354
[ 944.212279] 9e80: c04ab2fc c04aa344 df993200 df1fd800 00000100 df1fdbfc 00000000 df993200
[ 944.220473] 9ea0: 00000100 c04ab360 c0768000 c0023cf4 c0fba140 c03822b0 df1fdbfc 00000040
[ 944.228666] 9ec0: 0000012c c07abb40 c07abb48 c0778368 c07abb40 c0451288 c0769ed8 0000fb7d
[ 944.236859] 9ee0: c0fba140 00000001 0000000c c07ac850 c07ac840 c0768000 00000003 00000100
[ 944.245052] 9f00: 0000000c c001ebd4 00000000 c004782c c0fba140 0000000a 0000fb7c 00200000
[ 944.253245] 9f20: c0769f78 c078af20 00000018 00000000 c0769f78 c07abc01 00000001 c07abc01
[ 944.261438] 9f40: 00000000 c001ef18 c078af20 c000ea44 c07f6f40 000003ff c07f6f40 c00084d8
[ 944.269632] 9f60: c000ebc0 c0041c54 600f0013 ffffffff c0769fac c00110c0 00000000 00000000
[ 944.277825] 9f80: 00000000 c0779a48 c0768000 c0768000 c0768000 c07700cc c07abc01 00000001
[ 944.286018] 9fa0: c07abc01 00000000 01000000 c0769fc0 c000ebc0 c0041c54 600f0013 ffffffff
[ 944.294211] 9fc0: c0761530 c073ba14 ffffffff ffffffff c073b4f0 00000000 00000000 c0761530
[ 944.302405] 9fe0: 00000000 10c53c7d c0770078 c076152c c07765f8 00008070 00000000 00000000
[ 944.310608] [<c04b2094>] (tcp_gso_segment+0x1d8/0x390) from [<c04beeb4>] (inet_gso_segment+0x118/0x2dc)
[ 944.320029] [<c04beeb4>] (inet_gso_segment+0x118/0x2dc) from [<c0452dcc>] (skb_mac_gso_segment+0xa4/0x178)
[ 944.329704] [<c0452dcc>] (skb_mac_gso_segment+0xa4/0x178) from [<c04530e8>] (dev_hard_start_xmit+0x170/0x484)
[ 944.339643] [<c04530e8>] (dev_hard_start_xmit+0x170/0x484) from [<c0469af4>] (sch_direct_xmit+0xa4/0x19c)
[ 944.349231] [<c0469af4>] (sch_direct_xmit+0xa4/0x19c) from [<c04535b8>] (__dev_queue_xmit+0x1bc/0x3dc)
[ 944.358560] [<c04535b8>] (__dev_queue_xmit+0x1bc/0x3dc) from [<c0492604>] (ip_finish_output+0x1f4/0x440)
[ 944.368060] [<c0492604>] (ip_finish_output+0x1f4/0x440) from [<c0492a2c>] (ip_local_out+0x28/0x2c)
[ 944.377037] [<c0492a2c>] (ip_local_out+0x28/0x2c) from [<c0492b5c>] (ip_queue_xmit+0x12c/0x384)
[ 944.385754] [<c0492b5c>] (ip_queue_xmit+0x12c/0x384) from [<c04a76e8>] (tcp_transmit_skb+0x42c/0x86c)
[ 944.394992] [<c04a76e8>] (tcp_transmit_skb+0x42c/0x86c) from [<c04a7d68>] (tcp_write_xmit+0x174/0xa74)
[ 944.404317] [<c04a7d68>] (tcp_write_xmit+0x174/0xa74) from [<c04a86d8>] (__tcp_push_pending_frames+0x30/0x98)
[ 944.414252] [<c04a86d8>] (__tcp_push_pending_frames+0x30/0x98) from [<c04a4954>] (tcp_rcv_established+0x334/0x5a0)
[ 944.424622] [<c04a4954>] (tcp_rcv_established+0x334/0x5a0) from [<c04ac61c>] (tcp_v4_do_rcv+0x104/0x240)
[ 944.434122] [<c04ac61c>] (tcp_v4_do_rcv+0x104/0x240) from [<c04aede0>] (tcp_v4_rcv+0x6ec/0x728)
[ 944.442838] [<c04aede0>] (tcp_v4_rcv+0x6ec/0x728) from [<c048dab0>] (ip_local_deliver_finish+0x94/0x21c)
[ 944.452337] [<c048dab0>] (ip_local_deliver_finish+0x94/0x21c) from [<c048d85c>] (ip_rcv_finish+0x128/0x2e8)
[ 944.462098] [<c048d85c>] (ip_rcv_finish+0x128/0x2e8) from [<c044f018>] (__netif_receive_skb_core+0x4c4/0x5d0)
[ 944.472032] [<c044f018>] (__netif_receive_skb_core+0x4c4/0x5d0) from [<c045153c>] (napi_gro_receive+0x74/0xa0)
[ 944.482054] [<c045153c>] (napi_gro_receive+0x74/0xa0) from [<c0382238>] (mvneta_rx+0x420/0x498)
[ 944.490771] [<c0382238>] (mvneta_rx+0x420/0x498) from [<c0382354>] (mvneta_poll+0xa4/0x3b8)
[ 944.499139] [<c0382354>] (mvneta_poll+0xa4/0x3b8) from [<c0451288>] (net_rx_action+0x98/0x180)
[ 944.507773] [<c0451288>] (net_rx_action+0x98/0x180) from [<c001ebd4>] (__do_softirq+0xc8/0x1f4)
[ 944.516491] [<c001ebd4>] (__do_softirq+0xc8/0x1f4) from [<c001ef18>] (irq_exit+0x6c/0xa8)
[ 944.524690] [<c001ef18>] (irq_exit+0x6c/0xa8) from [<c000ea44>] (handle_IRQ+0x34/0x84)
[ 944.532624] [<c000ea44>] (handle_IRQ+0x34/0x84) from [<c00084d8>] (armada_370_xp_handle_irq+0x48/0x50)
[ 944.541953] [<c00084d8>] (armada_370_xp_handle_irq+0x48/0x50) from [<c00110c0>] (__irq_svc+0x40/0x50)
[ 944.551186] Exception stack(0xc0769f78 to 0xc0769fc0)
[ 944.556246] 9f60: 00000000 00000000
[ 944.564439] 9f80: 00000000 c0779a48 c0768000 c0768000 c0768000 c07700cc c07abc01 00000001
[ 944.572632] 9fa0: c07abc01 00000000 01000000 c0769fc0 c000ebc0 c0041c54 600f0013 ffffffff
[ 944.580835] [<c00110c0>] (__irq_svc+0x40/0x50) from [<c0041c54>] (cpu_startup_entry+0x44/0xe0)
[ 944.589470] [<c0041c54>] (cpu_startup_entry+0x44/0xe0) from [<c073ba14>] (start_kernel+0x2fc/0x358)
[ 944.598533] Code: e5942010 e5872010 e5977000 e59d3004 (e1d729b0)
[ 944.604662] ---[ end trace 39b798f37a10efc0 ]---
[ 944.609288] Kernel panic - not syncing: Fatal exception in interrupt
Here is the beginning of the assembly dump of tcp_gso_segment() provided by
arm-linux-gnueabi-objdump -S -EL -D -b binary -m arm --start-address=0x4a9ebc
--stop-address=0x4aa24c Image. If you wonder, LOADADDR is 0x8000 during
compilation, which explains the address shift.
004a9ebc <.data+0x4a9ebc>:
4a9ebc: e92d4ff0 push {r4, r5, r6, r7, r8, r9, sl, fp, lr}
4a9ec0: e1a09003 mov r9, r3
4a9ec4: e5901054 ldr r1, [r0, #84] ; 0x54
4a9ec8: e24dd014 sub sp, sp, #20
4a9ecc: e5903050 ldr r3, [r0, #80] ; 0x50
4a9ed0: e1a04000 mov r4, r0
4a9ed4: e1a08002 mov r8, r2
4a9ed8: e0611003 rsb r1, r1, r3
4a9edc: e3510013 cmp r1, #19
4a9ee0: 9a000099 bls 0x4aa14c
4a9ee4: e1d439b0 ldrh r3, [r4, #144] ; 0x90
4a9ee8: e59420a0 ldr r2, [r4, #160] ; 0xa0
4a9eec: e0823003 add r3, r2, r3
4a9ef0: e5d3500c ldrb r5, [r3, #12]
4a9ef4: e1a05225 lsr r5, r5, #4
4a9ef8: e1a05105 lsl r5, r5, #2
4a9efc: e3550013 cmp r5, #19
4a9f00: 9a000097 bls 0x4aa164
4a9f04: e5946050 ldr r6, [r4, #80] ; 0x50
4a9f08: e5943054 ldr r3, [r4, #84] ; 0x54
4a9f0c: e0631006 rsb r1, r3, r6
4a9f10: e1550001 cmp r5, r1
4a9f14: 8a0000c0 bhi 0x4aa21c
4a9f18: e065c006 rsb ip, r5, r6
4a9f1c: e584c050 str ip, [r4, #80] ; 0x50
4a9f20: e15c0003 cmp ip, r3
4a9f24: 3a0000c6 bcc 0x4aa244
4a9f28: e594709c ldr r7, [r4, #156] ; 0x9c
4a9f2c: e59430a4 ldr r3, [r4, #164] ; 0xa4
4a9f30: e0833005 add r3, r3, r5
4a9f34: e58430a4 str r3, [r4, #164] ; 0xa4
4a9f38: e1d700b2 ldrh r0, [r7, #2]
4a9f3c: e15c0000 cmp ip, r0
4a9f40: e58d0004 str r0, [sp, #4]
4a9f44: 9a000086 bls 0x4aa164
4a9f48: e1d710b6 ldrh r1, [r7, #6]
4a9f4c: e3a02701 mov r2, #262144 ; 0x40000
4a9f50: e3a03000 mov r3, #0
4a9f54: e1822008 orr r2, r2, r8
4a9f58: e1833009 orr r3, r3, r9
4a9f5c: e1a00801 lsl r0, r1, #16
4a9f60: e0022000 and r2, r2, r0
4a9f64: e1a0bfc0 asr fp, r0, #31
4a9f68: e1a0a000 mov sl, r0
4a9f6c: e003300b and r3, r3, fp
4a9f70: e15b0003 cmp fp, r3
4a9f74: 015a0002 cmpeq sl, r2
4a9f78: 0a00007d beq 0x4aa174
4a9f7c: e5d4107e ldrb r1, [r4, #126] ; 0x7e
4a9f80: e1a03009 mov r3, r9
4a9f84: e1a00004 mov r0, r4
4a9f88: e1a02008 mov r2, r8
4a9f8c: e7e071d1 ubfx r7, r1, #3, #1
4a9f90: e7c3119f bfc r1, #3, #1
4a9f94: e5c4107e strb r1, [r4, #126] ; 0x7e
4a9f98: e594b068 ldr fp, [r4, #104] ; 0x68
4a9f9c: ebfe59b7 bl 0x440680
4a9fa0: e3700a01 cmn r0, #4096 ; 0x1000
4a9fa4: e1a09000 mov r9, r0
4a9fa8: 8a00006e bhi 0x4aa168
4a9fac: e1d929b0 ldrh r2, [r9, #144] ; 0x90
4a9fb0: e1e06006 mvn r6, r6
4a9fb4: e59980a0 ldr r8, [r9, #160] ; 0xa0
4a9fb8: e6ff6076 uxth r6, r6
4a9fbc: e58d6008 str r6, [sp, #8]
4a9fc0: e5d9307e ldrb r3, [r9, #126] ; 0x7e
4a9fc4: e0888002 add r8, r8, r2
4a9fc8: e59f2278 ldr r2, [pc, #632] ; 0x4aa248
4a9fcc: e7c33197 bfi r3, r7, #3, #1
4a9fd0: e59d0004 ldr r0, [sp, #4]
4a9fd4: e5c9307e strb r3, [r9, #126] ; 0x7e
4a9fd8: e05b2002 subs r2, fp, r2
4a9fdc: e1d831b0 ldrh r3, [r8, #16]
4a9fe0: e272b000 rsbs fp, r2, #0
4a9fe4: e59d1008 ldr r1, [sp, #8]
4a9fe8: e085c000 add ip, r5, r0
4a9fec: e0bbb002 adcs fp, fp, r2
4a9ff0: e1a07009 mov r7, r9
4a9ff4: e08cc001 add ip, ip, r1
4a9ff8: e58d900c str r9, [sp, #12]
4a9ffc: e6bfcf3c rev ip, ip
4aa000: e08cc003 add ip, ip, r3
4aa004: e3a03000 mov r3, #0
4aa008: e08cc86c add ip, ip, ip, ror #16
4aa00c: e1a0c82c lsr ip, ip, #16
4aa010: e1a09003 mov r9, r3
4aa014: e5986004 ldr r6, [r8, #4]
4aa018: e1a0a00c mov sl, ip
4aa01c: e6bf6f36 rev r6, r6
4aa020: e5d8200d ldrb r2, [r8, #13]
4aa024: e1c8a1b0 strh sl, [r8, #16]
4aa028: e20220f7 and r2, r2, #247 ; 0xf7
4aa02c: e7c0201f bfc r2, #0, #1
4aa030: e5c8200d strb r2, [r8, #13]
4aa034: e5d72064 ldrb r2, [r7, #100] ; 0x64
4aa038: e202200c and r2, r2, #12
4aa03c: e352000c cmp r2, #12
4aa040: 0a000009 beq 0x4aa06c
4aa044: e1d709b0 ldrh r0, [r7, #144] ; 0x90
4aa048: e1a01005 mov r1, r5
4aa04c: e597e0a0 ldr lr, [r7, #160] ; 0xa0
4aa050: e597205c ldr r2, [r7, #92] ; 0x5c
4aa054: e08e0000 add r0, lr, r0
4aa058: ebf7cc4c bl 0x29d190
4aa05c: e0800860 add r0, r0, r0, ror #16
4aa060: e1e02000 mvn r2, r0
4aa064: e1a02822 lsr r2, r2, #16
4aa068: e1c821b0 strh r2, [r8, #16]
4aa06c: e35b0000 cmp fp, #0
4aa070: 0a000005 beq 0x4aa08c
4aa074: e5941068 ldr r1, [r4, #104] ; 0x68
4aa078: e59720a8 ldr r2, [r7, #168] ; 0xa8
4aa07c: e5871068 str r1, [r7, #104] ; 0x68
4aa080: e0899002 add r9, r9, r2
4aa084: e5942010 ldr r2, [r4, #16]
4aa088: e5872010 str r2, [r7, #16]
4aa08c: e5977000 ldr r7, [r7]
4aa090: e59d3004 ldr r3, [sp, #4]
4aa094: e1d729b0 ldrh r2, [r7, #144] ; 0x90 <- HERE!
4aa098: e0866003 add r6, r6, r3
4aa09c: e59780a0 ldr r8, [r7, #160] ; 0xa0
4aa0a0: e6bf1f36 rev r1, r6
4aa0a4: e0888002 add r8, r8, r2
4aa0a8: e5d8200d ldrb r2, [r8, #13]
4aa0ac: e5881004 str r1, [r8, #4]
4aa0b0: e7c7239f bfc r2, #7, #1
4aa0b4: e5c8200d strb r2, [r8, #13]
4aa0b8: e5972000 ldr r2, [r7]
4aa0bc: e3520000 cmp r2, #0
4aa0c0: 1affffd6 bne 0x4aa020
The line marked with the HERE! above is where the null pointer derefence
occurs. The code is in fact the (inlined) code of tcp_hdr(), which is in
turn the inlined code of tcp_transport_header() on given skb:
static inline unsigned char *skb_transport_header(const struct sk_buff *skb)
{
return skb->head + skb->transport_header;
}
The call to tcp_hdr() is in the following loop in tcp_gso_segment():
...
1 do {
2 th->fin = th->psh = 0;
3 th->check = newcheck;
4
5 if (skb->ip_summed != CHECKSUM_PARTIAL)
6 th->check =
7 csum_fold(csum_partial(skb_transport_header(skb),
8 thlen, skb->csum));
9
10 seq += mss;
11 if (copy_destructor) {
12 skb->destructor = gso_skb->destructor;
13 skb->sk = gso_skb->sk;
14 sum_truesize += skb->truesize;
15 }
16 skb = skb->next;
17 th = tcp_hdr(skb); <-- HERE!
18
19 th->seq = htonl(seq);
20 th->cwr = 0;
21 } while (skb->next);
...
Unless there is an assumption I missed somewhere in the function, the
problem may occur during the first round of the loop, because (unlike
the 'while' condition does at line 21) skb->next is not checked against
null at lines 17 above before it is passed to tcp_hdr() at line 18.
To be honest, I am asking because I am not familiar w/ the code and it
is somewhat old so I wonder why noone got hit before. AFAICT,
f4c50d990dcf ([NET]: Add software TSOv4) added TSOv4 support in 2006 via
introduction of tcp_tso_segmen() (with the same kind of deref but
possibly different assumptions) which was more recently modified via
28850dc7c7 (net: tcp: move GRO/GSO functions to tcp_offload) to become
tcp_gso_segment().
David, can you confirm the analysis and possibly comment on the
conditions needed for the bug to manifest?
Cheers,
a+
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] null pointer dereference in tcp_gso_segment()
2014-01-22 21:46 [BUG] null pointer dereference in tcp_gso_segment() Arnaud Ebalard
@ 2014-01-22 21:57 ` Eric Dumazet
2014-01-22 22:02 ` Arnaud Ebalard
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2014-01-22 21:57 UTC (permalink / raw)
To: Arnaud Ebalard
Cc: David Miller, Eric Dumazet, Daniel Borkmann, Herbert Xu,
Willy Tarreau, netdev
On Wed, 2014-01-22 at 22:46 +0100, Arnaud Ebalard wrote:
> Hi,
>
> The following is a backtrace I got while doing a simple transfer from my
> NAS (ARMv7-based ReadyNAS 102) running a 3.13.0 kernel. Usage was a
> simple file served by nginx. I managed to get a second identical one
> in another session. As this did not looked like some random race
> condition or some missing lock, I decided to take a look. Details are
> given below. FWIW, driver is mvneta (w/ fixes and performance patches
> you accepted for v3.14, David) and ethtool reports gso is enabled and tso
> disabled.
>
> [ 943.860383] Unable to handle kernel NULL pointer dereference at virtual address 00000090
> [ 943.868497] pgd = c0004000
> [ 943.871217] [00000090] *pgd=00000000
> [ 943.874813] Internal error: Oops: 17 [#1] ARM
> [ 943.879175] Modules linked in:
> [ 943.882248] CPU: 0 PID: 0 Comm: swapper Not tainted 3.13.0.rn102-00594-ga0fa1dd3cdbc-dirty #63
> [ 943.890873] task: c0775620 ti: c0768000 task.ti: c0768000
> [ 943.896291] PC is at tcp_gso_segment+0x1d8/0x390
> [ 943.900925] LR is at skb_segment+0x510/0x788
> [ 943.905202] pc : [<c04b2094>] lr : [<c0448b90>] psr: 200f0113
> [ 943.905202] sp : c0769aa8 ip : 00005b27 fp : 00000001
> [ 943.916697] r10: 00005b27 r9 : 00000868 r8 : dfab5cb0
> [ 943.921930] r7 : 00000000 r6 : 4ef79279 r5 : 00000020 r4 : deeba830
> [ 943.928467] r3 : 000005a8 r2 : df993200 r1 : c04a63f8 r0 : 000005a8
> [ 943.935005] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
> [ 943.942325] Control: 10c5387d Table: 1534c019 DAC: 00000015
> [ 943.948079] Process swapper (pid: 0, stack limit = 0xc0768238)
> [ 943.953921] Stack: (0xc0769aa8 to 0xc076a000)
> [ 943.958288] 9aa0: c077000c 000005a8 0000e397 d52f3540 c0771230 00001c48
> [ 943.966481] 9ac0: 00000006 deeba830 0000c6bc 00000014 0000009c 00000000 00000014 c04beeb4
> [ 943.974674] 9ae0: c0fbb8ec dec2ab80 0000008e 00000000 00000001 00004803 00000002 deeba830
> [ 943.982867] 9b00: c07739cc deeba830 df27d15c df1fd800 c05a045c c0452dcc 00000004 c07739e0
> [ 943.991061] 9b20: 00004803 00004803 00000002 00010000 00000000 deeba830 df27d15c c04530e8
> [ 943.999254] 9b40: 00000001 c046fc40 c0492410 df1fd800 00000000 00000000 00000000 def06e00
> [ 944.007447] 9b60: df1fd800 deeba830 df1fd800 df27d15c 00000010 c0469af4 def06e00 00000000
> [ 944.015640] 9b80: 00000042 00000000 df27d15c df1fd800 deeba830 c04535b8 def06e60 000333c5
> [ 944.023834] 9ba0: 00000000 dfaf5180 dfaf51fc 0000000e 00000000 deeba830 df1fd800 def35580
> [ 944.032027] 9bc0: 00000010 c0492604 80000000 0b50a8c0 00000000 deeba830 df993200 00000000
> [ 944.040221] 9be0: df993404 def35580 00000000 009c0000 00000000 c0492a2c deeba830 c0492b5c
> [ 944.048414] 9c00: deeba780 df993200 deeba780 00000020 00000000 0000fb7b 00000020 df993200
> [ 944.056608] 9c20: deeba830 c0778368 00000000 def35580 00000000 c04a76e8 c3a0f5c7 134b291d
> [ 944.064802] 9c40: 00000001 00000000 00000002 00000000 00000000 0000fb7b 002411ee 00000000
> [ 944.072995] 9c60: 79e31870 df993200 deeba780 000005a8 4ef79279 00001c48 00000000 000021f0
> [ 944.081188] 9c80: 00000000 c04a7d68 c39c239b 134b291d 00000006 000016a0 000021f0 00000001
> [ 944.089381] 9ca0: 00000000 df9932bc 00000002 00000000 00000006 134b291d d52f3540 df993200
> [ 944.097575] 9cc0: d567e7e4 d52f3540 00000020 d567e7d0 d52f3540 0000ddc7 c0770618 c04a86d8
> [ 944.105768] 9ce0: 00000020 df993200 df993200 c04a4954 00000000 00000020 00000001 00000002
> [ 944.113961] 9d00: d52f3540 df993200 def35a00 df993200 d567e7d0 d52f3540 0000ddc7 c04ac61c
> [ 944.122154] 9d20: c048da1c c046fc40 c048da1c c046fc40 dec12140 c0770b64 d52f3540 d52f3540
> [ 944.130348] 9d40: c07a3dc0 00000000 df993200 c04aede0 df1fd800 c046fcc0 00000000 c0769d74
> [ 944.138541] 9d60: c048da1c 80000000 c048d734 c0770b64 0000000e c0770b64 3050a8c0 c05aacb4
> [ 944.146734] 9d80: c0773470 00000000 c07a3dc0 d52f3540 d52f3540 00000000 c0770618 c048dab0
> [ 944.154927] 9da0: d567e7d0 c07739e8 c0770628 df1fd800 d52f3540 c048d85c df1b4400 c0770614
> [ 944.163120] 9dc0: c0770614 c07739e8 c0770628 df1fd800 00000008 c044f018 00000003 00000000
> [ 944.171314] 9de0: e1424800 42000000 df1fd800 d52f3540 00000001 c0770628 00000000 d52f3540
> [ 944.179507] 9e00: 00000003 df3bd400 e1424800 42000000 df1fd800 d52f3540 00000001 c045153c
> [ 944.187700] 9e20: 00000004 c0019430 e1424000 c0382238 00000000 00000000 00000002 dfbc4600
> [ 944.195893] 9e40: 00000001 df1fdbe0 00000001 00000042 00000000 00000000 dee34000 00000040
> [ 944.204086] 9e60: df1fdbe0 00000100 00000000 00000100 df3bd400 c079b684 00000000 c0382354
> [ 944.212279] 9e80: c04ab2fc c04aa344 df993200 df1fd800 00000100 df1fdbfc 00000000 df993200
> [ 944.220473] 9ea0: 00000100 c04ab360 c0768000 c0023cf4 c0fba140 c03822b0 df1fdbfc 00000040
> [ 944.228666] 9ec0: 0000012c c07abb40 c07abb48 c0778368 c07abb40 c0451288 c0769ed8 0000fb7d
> [ 944.236859] 9ee0: c0fba140 00000001 0000000c c07ac850 c07ac840 c0768000 00000003 00000100
> [ 944.245052] 9f00: 0000000c c001ebd4 00000000 c004782c c0fba140 0000000a 0000fb7c 00200000
> [ 944.253245] 9f20: c0769f78 c078af20 00000018 00000000 c0769f78 c07abc01 00000001 c07abc01
> [ 944.261438] 9f40: 00000000 c001ef18 c078af20 c000ea44 c07f6f40 000003ff c07f6f40 c00084d8
> [ 944.269632] 9f60: c000ebc0 c0041c54 600f0013 ffffffff c0769fac c00110c0 00000000 00000000
> [ 944.277825] 9f80: 00000000 c0779a48 c0768000 c0768000 c0768000 c07700cc c07abc01 00000001
> [ 944.286018] 9fa0: c07abc01 00000000 01000000 c0769fc0 c000ebc0 c0041c54 600f0013 ffffffff
> [ 944.294211] 9fc0: c0761530 c073ba14 ffffffff ffffffff c073b4f0 00000000 00000000 c0761530
> [ 944.302405] 9fe0: 00000000 10c53c7d c0770078 c076152c c07765f8 00008070 00000000 00000000
> [ 944.310608] [<c04b2094>] (tcp_gso_segment+0x1d8/0x390) from [<c04beeb4>] (inet_gso_segment+0x118/0x2dc)
> [ 944.320029] [<c04beeb4>] (inet_gso_segment+0x118/0x2dc) from [<c0452dcc>] (skb_mac_gso_segment+0xa4/0x178)
> [ 944.329704] [<c0452dcc>] (skb_mac_gso_segment+0xa4/0x178) from [<c04530e8>] (dev_hard_start_xmit+0x170/0x484)
> [ 944.339643] [<c04530e8>] (dev_hard_start_xmit+0x170/0x484) from [<c0469af4>] (sch_direct_xmit+0xa4/0x19c)
> [ 944.349231] [<c0469af4>] (sch_direct_xmit+0xa4/0x19c) from [<c04535b8>] (__dev_queue_xmit+0x1bc/0x3dc)
> [ 944.358560] [<c04535b8>] (__dev_queue_xmit+0x1bc/0x3dc) from [<c0492604>] (ip_finish_output+0x1f4/0x440)
> [ 944.368060] [<c0492604>] (ip_finish_output+0x1f4/0x440) from [<c0492a2c>] (ip_local_out+0x28/0x2c)
> [ 944.377037] [<c0492a2c>] (ip_local_out+0x28/0x2c) from [<c0492b5c>] (ip_queue_xmit+0x12c/0x384)
> [ 944.385754] [<c0492b5c>] (ip_queue_xmit+0x12c/0x384) from [<c04a76e8>] (tcp_transmit_skb+0x42c/0x86c)
> [ 944.394992] [<c04a76e8>] (tcp_transmit_skb+0x42c/0x86c) from [<c04a7d68>] (tcp_write_xmit+0x174/0xa74)
> [ 944.404317] [<c04a7d68>] (tcp_write_xmit+0x174/0xa74) from [<c04a86d8>] (__tcp_push_pending_frames+0x30/0x98)
> [ 944.414252] [<c04a86d8>] (__tcp_push_pending_frames+0x30/0x98) from [<c04a4954>] (tcp_rcv_established+0x334/0x5a0)
> [ 944.424622] [<c04a4954>] (tcp_rcv_established+0x334/0x5a0) from [<c04ac61c>] (tcp_v4_do_rcv+0x104/0x240)
> [ 944.434122] [<c04ac61c>] (tcp_v4_do_rcv+0x104/0x240) from [<c04aede0>] (tcp_v4_rcv+0x6ec/0x728)
> [ 944.442838] [<c04aede0>] (tcp_v4_rcv+0x6ec/0x728) from [<c048dab0>] (ip_local_deliver_finish+0x94/0x21c)
> [ 944.452337] [<c048dab0>] (ip_local_deliver_finish+0x94/0x21c) from [<c048d85c>] (ip_rcv_finish+0x128/0x2e8)
> [ 944.462098] [<c048d85c>] (ip_rcv_finish+0x128/0x2e8) from [<c044f018>] (__netif_receive_skb_core+0x4c4/0x5d0)
> [ 944.472032] [<c044f018>] (__netif_receive_skb_core+0x4c4/0x5d0) from [<c045153c>] (napi_gro_receive+0x74/0xa0)
> [ 944.482054] [<c045153c>] (napi_gro_receive+0x74/0xa0) from [<c0382238>] (mvneta_rx+0x420/0x498)
> [ 944.490771] [<c0382238>] (mvneta_rx+0x420/0x498) from [<c0382354>] (mvneta_poll+0xa4/0x3b8)
> [ 944.499139] [<c0382354>] (mvneta_poll+0xa4/0x3b8) from [<c0451288>] (net_rx_action+0x98/0x180)
> [ 944.507773] [<c0451288>] (net_rx_action+0x98/0x180) from [<c001ebd4>] (__do_softirq+0xc8/0x1f4)
> [ 944.516491] [<c001ebd4>] (__do_softirq+0xc8/0x1f4) from [<c001ef18>] (irq_exit+0x6c/0xa8)
> [ 944.524690] [<c001ef18>] (irq_exit+0x6c/0xa8) from [<c000ea44>] (handle_IRQ+0x34/0x84)
> [ 944.532624] [<c000ea44>] (handle_IRQ+0x34/0x84) from [<c00084d8>] (armada_370_xp_handle_irq+0x48/0x50)
> [ 944.541953] [<c00084d8>] (armada_370_xp_handle_irq+0x48/0x50) from [<c00110c0>] (__irq_svc+0x40/0x50)
> [ 944.551186] Exception stack(0xc0769f78 to 0xc0769fc0)
> [ 944.556246] 9f60: 00000000 00000000
> [ 944.564439] 9f80: 00000000 c0779a48 c0768000 c0768000 c0768000 c07700cc c07abc01 00000001
> [ 944.572632] 9fa0: c07abc01 00000000 01000000 c0769fc0 c000ebc0 c0041c54 600f0013 ffffffff
> [ 944.580835] [<c00110c0>] (__irq_svc+0x40/0x50) from [<c0041c54>] (cpu_startup_entry+0x44/0xe0)
> [ 944.589470] [<c0041c54>] (cpu_startup_entry+0x44/0xe0) from [<c073ba14>] (start_kernel+0x2fc/0x358)
> [ 944.598533] Code: e5942010 e5872010 e5977000 e59d3004 (e1d729b0)
> [ 944.604662] ---[ end trace 39b798f37a10efc0 ]---
> [ 944.609288] Kernel panic - not syncing: Fatal exception in interrupt
>
>
> Here is the beginning of the assembly dump of tcp_gso_segment() provided by
> arm-linux-gnueabi-objdump -S -EL -D -b binary -m arm --start-address=0x4a9ebc
> --stop-address=0x4aa24c Image. If you wonder, LOADADDR is 0x8000 during
> compilation, which explains the address shift.
> The line marked with the HERE! above is where the null pointer derefence
> occurs. The code is in fact the (inlined) code of tcp_hdr(), which is in
> turn the inlined code of tcp_transport_header() on given skb:
>
> static inline unsigned char *skb_transport_header(const struct sk_buff *skb)
> {
> return skb->head + skb->transport_header;
> }
>
> The call to tcp_hdr() is in the following loop in tcp_gso_segment():
>
> ...
> 1 do {
> 2 th->fin = th->psh = 0;
> 3 th->check = newcheck;
> 4
> 5 if (skb->ip_summed != CHECKSUM_PARTIAL)
> 6 th->check =
> 7 csum_fold(csum_partial(skb_transport_header(skb),
> 8 thlen, skb->csum));
> 9
> 10 seq += mss;
> 11 if (copy_destructor) {
> 12 skb->destructor = gso_skb->destructor;
> 13 skb->sk = gso_skb->sk;
> 14 sum_truesize += skb->truesize;
> 15 }
> 16 skb = skb->next;
> 17 th = tcp_hdr(skb); <-- HERE!
> 18
> 19 th->seq = htonl(seq);
> 20 th->cwr = 0;
> 21 } while (skb->next);
> ...
>
> Unless there is an assumption I missed somewhere in the function, the
> problem may occur during the first round of the loop, because (unlike
> the 'while' condition does at line 21) skb->next is not checked against
> null at lines 17 above before it is passed to tcp_hdr() at line 18.
>
> To be honest, I am asking because I am not familiar w/ the code and it
> is somewhat old so I wonder why noone got hit before. AFAICT,
> f4c50d990dcf ([NET]: Add software TSOv4) added TSOv4 support in 2006 via
> introduction of tcp_tso_segmen() (with the same kind of deref but
> possibly different assumptions) which was more recently modified via
> 28850dc7c7 (net: tcp: move GRO/GSO functions to tcp_offload) to become
> tcp_gso_segment().
>
> David, can you confirm the analysis and possibly comment on the
> conditions needed for the bug to manifest?
A gso packet contains at least 2 segments.
So the NULL deref should not happen.
Something strange is happening here...
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] null pointer dereference in tcp_gso_segment()
2014-01-22 21:57 ` Eric Dumazet
@ 2014-01-22 22:02 ` Arnaud Ebalard
2014-01-22 22:18 ` Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Arnaud Ebalard @ 2014-01-22 22:02 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, Eric Dumazet, Daniel Borkmann, Herbert Xu,
Willy Tarreau, netdev
Hi Eric,
Eric Dumazet <eric.dumazet@gmail.com> writes:
>> Unless there is an assumption I missed somewhere in the function, the
>> problem may occur during the first round of the loop, because (unlike
>> the 'while' condition does at line 21) skb->next is not checked against
>> null at lines 17 above before it is passed to tcp_hdr() at line 18.
>>
>> To be honest, I am asking because I am not familiar w/ the code and it
>> is somewhat old so I wonder why noone got hit before. AFAICT,
>> f4c50d990dcf ([NET]: Add software TSOv4) added TSOv4 support in 2006 via
>> introduction of tcp_tso_segmen() (with the same kind of deref but
>> possibly different assumptions) which was more recently modified via
>> 28850dc7c7 (net: tcp: move GRO/GSO functions to tcp_offload) to become
>> tcp_gso_segment().
>>
>> David, can you confirm the analysis and possibly comment on the
>> conditions needed for the bug to manifest?
>
> A gso packet contains at least 2 segments.
By whom / where is it enforced?
Cheers,
a+
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] null pointer dereference in tcp_gso_segment()
2014-01-22 22:02 ` Arnaud Ebalard
@ 2014-01-22 22:18 ` Eric Dumazet
2014-01-22 23:56 ` Willy Tarreau
2014-01-25 23:54 ` Arnaud Ebalard
0 siblings, 2 replies; 9+ messages in thread
From: Eric Dumazet @ 2014-01-22 22:18 UTC (permalink / raw)
To: Arnaud Ebalard
Cc: David Miller, Eric Dumazet, Daniel Borkmann, Herbert Xu,
Willy Tarreau, netdev
On Wed, 2014-01-22 at 23:02 +0100, Arnaud Ebalard wrote:
> Hi Eric,
>
> Eric Dumazet <eric.dumazet@gmail.com> writes:
>
> >> Unless there is an assumption I missed somewhere in the function, the
> >> problem may occur during the first round of the loop, because (unlike
> >> the 'while' condition does at line 21) skb->next is not checked against
> >> null at lines 17 above before it is passed to tcp_hdr() at line 18.
> >>
> >> To be honest, I am asking because I am not familiar w/ the code and it
> >> is somewhat old so I wonder why noone got hit before. AFAICT,
> >> f4c50d990dcf ([NET]: Add software TSOv4) added TSOv4 support in 2006 via
> >> introduction of tcp_tso_segmen() (with the same kind of deref but
> >> possibly different assumptions) which was more recently modified via
> >> 28850dc7c7 (net: tcp: move GRO/GSO functions to tcp_offload) to become
> >> tcp_gso_segment().
> >>
> >> David, can you confirm the analysis and possibly comment on the
> >> conditions needed for the bug to manifest?
> >
> > A gso packet contains at least 2 segments.
>
> By whom / where is it enforced?
For example, tcp_gso_segment() does the following check :
if (unlikely(skb->len <= mss))
goto out;
If there was one segment, then skb->len should also be smaller than mss
Since TCP stack seemed to be the provider of the packet in your stack
trace, check tcp_set_skb_tso_segs()
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] null pointer dereference in tcp_gso_segment()
2014-01-22 22:18 ` Eric Dumazet
@ 2014-01-22 23:56 ` Willy Tarreau
2014-01-26 0:04 ` Arnaud Ebalard
2014-01-25 23:54 ` Arnaud Ebalard
1 sibling, 1 reply; 9+ messages in thread
From: Willy Tarreau @ 2014-01-22 23:56 UTC (permalink / raw)
To: Eric Dumazet
Cc: Arnaud Ebalard, David Miller, Eric Dumazet, Daniel Borkmann,
Herbert Xu, netdev
On Wed, Jan 22, 2014 at 02:18:45PM -0800, Eric Dumazet wrote:
> On Wed, 2014-01-22 at 23:02 +0100, Arnaud Ebalard wrote:
> > Hi Eric,
> >
> > Eric Dumazet <eric.dumazet@gmail.com> writes:
> >
> > >> Unless there is an assumption I missed somewhere in the function, the
> > >> problem may occur during the first round of the loop, because (unlike
> > >> the 'while' condition does at line 21) skb->next is not checked against
> > >> null at lines 17 above before it is passed to tcp_hdr() at line 18.
> > >>
> > >> To be honest, I am asking because I am not familiar w/ the code and it
> > >> is somewhat old so I wonder why noone got hit before. AFAICT,
> > >> f4c50d990dcf ([NET]: Add software TSOv4) added TSOv4 support in 2006 via
> > >> introduction of tcp_tso_segmen() (with the same kind of deref but
> > >> possibly different assumptions) which was more recently modified via
> > >> 28850dc7c7 (net: tcp: move GRO/GSO functions to tcp_offload) to become
> > >> tcp_gso_segment().
> > >>
> > >> David, can you confirm the analysis and possibly comment on the
> > >> conditions needed for the bug to manifest?
> > >
> > > A gso packet contains at least 2 segments.
> >
> > By whom / where is it enforced?
>
> For example, tcp_gso_segment() does the following check :
>
> if (unlikely(skb->len <= mss))
> goto out;
>
> If there was one segment, then skb->len should also be smaller than mss
>
> Since TCP stack seemed to be the provider of the packet in your stack
> trace, check tcp_set_skb_tso_segs()
Thanks Eric for the explanation. From Arnaud's trace, I suspect that he's
received an ACK which has released some pending data, so it's very likely
indeed that at least two segments were released at once given that the
receiver is likely to ACK every two segments.
Also we can expect that the received ACK was copy-breaked. I don't know
if some sort of skb recycling may happen at this stage and reveal some
bad corner cases (eg: improperly initialized skb during the rx path that
causes everything to break when it's recycled for the tx path), but Arnaud
you can easily disable the rx_copybreak feature by setting the rx_copybreak
module argument to zero. You can change it at run time in /sys/module. At
least it will tell us if it could be related or not.
Regards,
Willy
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] null pointer dereference in tcp_gso_segment()
2014-01-22 22:18 ` Eric Dumazet
2014-01-22 23:56 ` Willy Tarreau
@ 2014-01-25 23:54 ` Arnaud Ebalard
2014-01-26 1:18 ` Eric Dumazet
1 sibling, 1 reply; 9+ messages in thread
From: Arnaud Ebalard @ 2014-01-25 23:54 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, Eric Dumazet, Daniel Borkmann, Herbert Xu,
Willy Tarreau, netdev
Hi Eric,
Eric Dumazet <eric.dumazet@gmail.com> writes:
> On Wed, 2014-01-22 at 23:02 +0100, Arnaud Ebalard wrote:
>> Hi Eric,
>>
>> Eric Dumazet <eric.dumazet@gmail.com> writes:
>>
>> >> Unless there is an assumption I missed somewhere in the function, the
>> >> problem may occur during the first round of the loop, because (unlike
>> >> the 'while' condition does at line 21) skb->next is not checked against
>> >> null at lines 17 above before it is passed to tcp_hdr() at line 18.
>> >>
>> >> To be honest, I am asking because I am not familiar w/ the code and it
>> >> is somewhat old so I wonder why noone got hit before. AFAICT,
>> >> f4c50d990dcf ([NET]: Add software TSOv4) added TSOv4 support in 2006 via
>> >> introduction of tcp_tso_segmen() (with the same kind of deref but
>> >> possibly different assumptions) which was more recently modified via
>> >> 28850dc7c7 (net: tcp: move GRO/GSO functions to tcp_offload) to become
>> >> tcp_gso_segment().
>> >>
>> >> David, can you confirm the analysis and possibly comment on the
>> >> conditions needed for the bug to manifest?
>> >
>> > A gso packet contains at least 2 segments.
>>
>> By whom / where is it enforced?
>
> For example, tcp_gso_segment() does the following check :
>
> if (unlikely(skb->len <= mss))
> goto out;
>
> If there was one segment, then skb->len should also be smaller than
> mss
Thanks for the explanation and sorry for the delay, I only just found
the time to take a look at the code. For the discussion, a simplified
version of tcp_gso_segment() is:
th = tcp_hdr(skb);
thlen = th->doff * 4;
...
__skb_pull(skb, thlen);
...
mss = tcp_skb_mss(skb);
if (unlikely(skb->len <= mss))
goto out;
...
segs = skb_segment(skb, features);
skb = segs;
...
skb = skb->next;
th = tcp_hdr(skb); <- bug occurs here
So the logic seems to be that if we pass the mss test (i.e. skb->len >
mss), then skb_segment() *should* indeed create at least two segments
from the skb. I took a look at skb_segment() but the code is !trivial,
i.e. it is not obvious that there is no way for the function to deliver
a sk_buff skb w/ a NULL skb->next. Eric, I guess you or Herbert are
familiar enough w/ the code to tell. But before checking that, your lead
below is interesting ...
> Since TCP stack seemed to be the provider of the packet in your stack
> trace, check tcp_set_skb_tso_segs()
It is indeed called in tcp_write_xmit() which appears in the
backtrace. That function you point has an interesting property:
static void tcp_set_skb_tso_segs(const struct sock *sk, struct sk_buff *skb,
unsigned int mss_now)
{
/* Make sure we own this skb before messing gso_size/gso_segs */
WARN_ON_ONCE(skb_cloned(skb));
if (skb->len <= mss_now || skb->ip_summed == CHECKSUM_NONE) {
/* Avoid the costly divide in the normal
* non-TSO case.
*/
skb_shinfo(skb)->gso_segs = 1;
skb_shinfo(skb)->gso_size = 0;
skb_shinfo(skb)->gso_type = 0;
} else {
skb_shinfo(skb)->gso_segs = DIV_ROUND_UP(skb->len, mss_now);
skb_shinfo(skb)->gso_size = mss_now;
skb_shinfo(skb)->gso_type = sk->sk_gso_type;
}
}
If it is called with skb->len <= mss, the resulting skb will be modified
so that you will then have skb_shinfo(skb)->gso_size set to 0,
i.e. skb->len > skb_shinfo(skb)->gso_size.
In tcp_gso_segment(), mss is grabbed using tcp_skb_mss() which simply
returns skb_shinfo(skb)->gso_size. That function comes with a comment
indicating that it provides the mss only when tcp_skb_pcount() > 1,
i.e when skb_shinfo(skb)->gso_segs > 1. Said differently, one should
never call tcp_skb_mss() after tcp_set_skb_tso_segs() has been called on
a skb *unless* she tests explicitly that tcp_skb_pcount() > 1.
This test (tcp_skb_pcount() > 1) is not done in tcp_gso_segment() before
getting the mss value w/ tcp_skb_mss(). I may have missed a test
somewhere in a caller but I do not see why tcp_gso_segment() makes the
assumption it can safely call tcp_skb_mss().
Cheers,
a+
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] null pointer dereference in tcp_gso_segment()
2014-01-22 23:56 ` Willy Tarreau
@ 2014-01-26 0:04 ` Arnaud Ebalard
0 siblings, 0 replies; 9+ messages in thread
From: Arnaud Ebalard @ 2014-01-26 0:04 UTC (permalink / raw)
To: Willy Tarreau
Cc: Eric Dumazet, David Miller, Eric Dumazet, Daniel Borkmann,
Herbert Xu, netdev
Hi,
Willy Tarreau <w@1wt.eu> writes:
> On Wed, Jan 22, 2014 at 02:18:45PM -0800, Eric Dumazet wrote:
>> On Wed, 2014-01-22 at 23:02 +0100, Arnaud Ebalard wrote:
>> > Hi Eric,
>> >
>> > Eric Dumazet <eric.dumazet@gmail.com> writes:
>> >
>> > >> Unless there is an assumption I missed somewhere in the function, the
>> > >> problem may occur during the first round of the loop, because (unlike
>> > >> the 'while' condition does at line 21) skb->next is not checked against
>> > >> null at lines 17 above before it is passed to tcp_hdr() at line 18.
>> > >>
>> > >> To be honest, I am asking because I am not familiar w/ the code and it
>> > >> is somewhat old so I wonder why noone got hit before. AFAICT,
>> > >> f4c50d990dcf ([NET]: Add software TSOv4) added TSOv4 support in 2006 via
>> > >> introduction of tcp_tso_segmen() (with the same kind of deref but
>> > >> possibly different assumptions) which was more recently modified via
>> > >> 28850dc7c7 (net: tcp: move GRO/GSO functions to tcp_offload) to become
>> > >> tcp_gso_segment().
>> > >>
>> > >> David, can you confirm the analysis and possibly comment on the
>> > >> conditions needed for the bug to manifest?
>> > >
>> > > A gso packet contains at least 2 segments.
>> >
>> > By whom / where is it enforced?
>>
>> For example, tcp_gso_segment() does the following check :
>>
>> if (unlikely(skb->len <= mss))
>> goto out;
>>
>> If there was one segment, then skb->len should also be smaller than mss
>>
>> Since TCP stack seemed to be the provider of the packet in your stack
>> trace, check tcp_set_skb_tso_segs()
>
> Thanks Eric for the explanation. From Arnaud's trace, I suspect that he's
> received an ACK which has released some pending data, so it's very likely
> indeed that at least two segments were released at once given that the
> receiver is likely to ACK every two segments.
>
> Also we can expect that the received ACK was copy-breaked. I don't know
> if some sort of skb recycling may happen at this stage and reveal some
> bad corner cases (eg: improperly initialized skb during the rx path that
> causes everything to break when it's recycled for the tx path), but Arnaud
> you can easily disable the rx_copybreak feature by setting the rx_copybreak
> module argument to zero. You can change it at run time in /sys/module. At
> least it will tell us if it could be related or not.
The problem is that I cannot simply use that trick to test your hypothesis
as the bug is not easily reproducible. I was lucky to trigger it twice but
never got it then (when I tested with an additonal BUG_ON(skb->next == NULL)
before the main loop in tcp_gso_segment()).
Cheers,
a+
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] null pointer dereference in tcp_gso_segment()
2014-01-25 23:54 ` Arnaud Ebalard
@ 2014-01-26 1:18 ` Eric Dumazet
2014-01-27 22:14 ` Arnaud Ebalard
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2014-01-26 1:18 UTC (permalink / raw)
To: Arnaud Ebalard
Cc: David Miller, Eric Dumazet, Daniel Borkmann, Herbert Xu,
Willy Tarreau, netdev
On Sun, 2014-01-26 at 00:54 +0100, Arnaud Ebalard wrote:
> Thanks for the explanation and sorry for the delay, I only just found
> the time to take a look at the code. For the discussion, a simplified
> version of tcp_gso_segment() is:
>
>
> th = tcp_hdr(skb);
> thlen = th->doff * 4;
>
> ...
>
> __skb_pull(skb, thlen);
>
> ...
>
> mss = tcp_skb_mss(skb);
> if (unlikely(skb->len <= mss))
> goto out;
>
> ...
>
> segs = skb_segment(skb, features);
> skb = segs;
>
> ...
>
> skb = skb->next;
> th = tcp_hdr(skb); <- bug occurs here
>
>
> So the logic seems to be that if we pass the mss test (i.e. skb->len >
> mss), then skb_segment() *should* indeed create at least two segments
> from the skb. I took a look at skb_segment() but the code is !trivial,
> i.e. it is not obvious that there is no way for the function to deliver
> a sk_buff skb w/ a NULL skb->next. Eric, I guess you or Herbert are
> familiar enough w/ the code to tell. But before checking that, your lead
> below is interesting ...
>
> > Since TCP stack seemed to be the provider of the packet in your stack
> > trace, check tcp_set_skb_tso_segs()
>
> It is indeed called in tcp_write_xmit() which appears in the
> backtrace. That function you point has an interesting property:
>
> static void tcp_set_skb_tso_segs(const struct sock *sk, struct sk_buff *skb,
> unsigned int mss_now)
> {
> /* Make sure we own this skb before messing gso_size/gso_segs */
> WARN_ON_ONCE(skb_cloned(skb));
>
> if (skb->len <= mss_now || skb->ip_summed == CHECKSUM_NONE) {
> /* Avoid the costly divide in the normal
> * non-TSO case.
> */
> skb_shinfo(skb)->gso_segs = 1;
> skb_shinfo(skb)->gso_size = 0;
> skb_shinfo(skb)->gso_type = 0;
> } else {
> skb_shinfo(skb)->gso_segs = DIV_ROUND_UP(skb->len, mss_now);
> skb_shinfo(skb)->gso_size = mss_now;
> skb_shinfo(skb)->gso_type = sk->sk_gso_type;
> }
> }
>
> If it is called with skb->len <= mss, the resulting skb will be modified
> so that you will then have skb_shinfo(skb)->gso_size set to 0,
> i.e. skb->len > skb_shinfo(skb)->gso_size.
>
The key part is that gso_size is set to 0.
Meaning its not a gso packet :
static inline bool skb_is_gso(const struct sk_buff *skb)
{
return skb_shinfo(skb)->gso_size;
}
So the packet cannot possibly go up to tcp_gso_segment() because
we fail the first test in :
static inline bool netif_needs_gso(struct sk_buff *skb,
netdev_features_t features)
{
return skb_is_gso(skb) && (!skb_gso_ok(skb, features) ||
unlikely((skb->ip_summed != CHECKSUM_PARTIAL) &&
(skb->ip_summed != CHECKSUM_UNNECESSARY)));
}
> In tcp_gso_segment(), mss is grabbed using tcp_skb_mss() which simply
> returns skb_shinfo(skb)->gso_size. That function comes with a comment
> indicating that it provides the mss only when tcp_skb_pcount() > 1,
> i.e when skb_shinfo(skb)->gso_segs > 1. Said differently, one should
> never call tcp_skb_mss() after tcp_set_skb_tso_segs() has been called on
> a skb *unless* she tests explicitly that tcp_skb_pcount() > 1.
>
> This test (tcp_skb_pcount() > 1) is not done in tcp_gso_segment() before
> getting the mss value w/ tcp_skb_mss(). I may have missed a test
> somewhere in a caller but I do not see why tcp_gso_segment() makes the
> assumption it can safely call tcp_skb_mss().
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] null pointer dereference in tcp_gso_segment()
2014-01-26 1:18 ` Eric Dumazet
@ 2014-01-27 22:14 ` Arnaud Ebalard
0 siblings, 0 replies; 9+ messages in thread
From: Arnaud Ebalard @ 2014-01-27 22:14 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, Eric Dumazet, Daniel Borkmann, Herbert Xu,
Willy Tarreau, netdev
Hi Eric,
Eric Dumazet <eric.dumazet@gmail.com> writes:
> On Sun, 2014-01-26 at 00:54 +0100, Arnaud Ebalard wrote:
>
>> Thanks for the explanation and sorry for the delay, I only just found
>> the time to take a look at the code. For the discussion, a simplified
>> version of tcp_gso_segment() is:
>>
>>
>> th = tcp_hdr(skb);
>> thlen = th->doff * 4;
>>
>> ...
>>
>> __skb_pull(skb, thlen);
>>
>> ...
>>
>> mss = tcp_skb_mss(skb);
>> if (unlikely(skb->len <= mss))
>> goto out;
>>
>> ...
>>
>> segs = skb_segment(skb, features);
>> skb = segs;
>>
>> ...
>>
>> skb = skb->next;
>> th = tcp_hdr(skb); <- bug occurs here
>>
>>
>> So the logic seems to be that if we pass the mss test (i.e. skb->len >
>> mss), then skb_segment() *should* indeed create at least two segments
>> from the skb. I took a look at skb_segment() but the code is !trivial,
>> i.e. it is not obvious that there is no way for the function to deliver
>> a sk_buff skb w/ a NULL skb->next. Eric, I guess you or Herbert are
>> familiar enough w/ the code to tell. But before checking that, your lead
>> below is interesting ...
>>
>> > Since TCP stack seemed to be the provider of the packet in your stack
>> > trace, check tcp_set_skb_tso_segs()
>>
>> It is indeed called in tcp_write_xmit() which appears in the
>> backtrace. That function you point has an interesting property:
>>
>> static void tcp_set_skb_tso_segs(const struct sock *sk, struct sk_buff *skb,
>> unsigned int mss_now)
>> {
>> /* Make sure we own this skb before messing gso_size/gso_segs */
>> WARN_ON_ONCE(skb_cloned(skb));
>>
>> if (skb->len <= mss_now || skb->ip_summed == CHECKSUM_NONE) {
>> /* Avoid the costly divide in the normal
>> * non-TSO case.
>> */
>> skb_shinfo(skb)->gso_segs = 1;
>> skb_shinfo(skb)->gso_size = 0;
>> skb_shinfo(skb)->gso_type = 0;
>> } else {
>> skb_shinfo(skb)->gso_segs = DIV_ROUND_UP(skb->len, mss_now);
>> skb_shinfo(skb)->gso_size = mss_now;
>> skb_shinfo(skb)->gso_type = sk->sk_gso_type;
>> }
>> }
>>
>> If it is called with skb->len <= mss, the resulting skb will be modified
>> so that you will then have skb_shinfo(skb)->gso_size set to 0,
>> i.e. skb->len > skb_shinfo(skb)->gso_size.
>>
>
> The key part is that gso_size is set to 0.
>
> Meaning its not a gso packet :
>
> static inline bool skb_is_gso(const struct sk_buff *skb)
> {
> return skb_shinfo(skb)->gso_size;
> }
>
> So the packet cannot possibly go up to tcp_gso_segment() because
> we fail the first test in :
>
> static inline bool netif_needs_gso(struct sk_buff *skb,
> netdev_features_t features)
> {
> return skb_is_gso(skb) && (!skb_gso_ok(skb, features) ||
> unlikely((skb->ip_summed != CHECKSUM_PARTIAL) &&
> (skb->ip_summed != CHECKSUM_UNNECESSARY)));
> }
Considering that, I decided to spend some time tonight on the other
possibility i.e. something happening in skb_segment() resulting in
skb->next being NULL. I monitored how the skb look like before entering
skb_segment(), how it looks after the end of the loop and also how many
round of the loop below do happen:
segs = skb_segment(skb, features);
if (IS_ERR(segs))
goto out;
/* Only first segment might have ooo_okay set */
segs->ooo_okay = ooo_okay;
delta = htonl(oldlen + (thlen + mss));
skb = segs;
th = tcp_hdr(skb);
seq = ntohl(th->seq);
newcheck = ~csum_fold((__force __wsum)((__force u32)th->check +
(__force u32)delta));
do {
th->fin = th->psh = 0;
th->check = newcheck;
if (skb->ip_summed != CHECKSUM_PARTIAL)
th->check =
csum_fold(csum_partial(skb_transport_header(skb),
thlen, skb->csum));
seq += mss;
if (copy_destructor) {
skb->destructor = gso_skb->destructor;
skb->sk = gso_skb->sk;
sum_truesize += skb->truesize;
}
skb = skb->next;
th = tcp_hdr(skb);
th->seq = htonl(seq);
th->cwr = 0;
} while (skb->next);
Usually, if we initially have tcp_skb_pcount(skb) == X, then we end up
doing X-1 rounds in the loop (i.e. skb is a list of X chained
sk_buff). But I managed to reproduce the two following situations in a
row in a same download session just before a crash):
before skb_segment(), tcp_skb_pcount(skb) is 17 but we only do 7 rounds
in the loop, i.e. the result is a list of 8 sk_buff.
before skb_segment(), tcp_skb_pcount(skb) is 14 but we only do 1 round
in the loop, i.e. the result is a list of 2 sk_buff.
Eric, if you still accept two potentially stupid questions:
- is skb_segment() behavior expected (e.g. frag or frag_list related)?
- if the above can happen, what prevents having skb_segment() deliver a
sk_buff w/ skb->next set to NULL?
The full trace of the crash is provided below. Note that the function
dev_queue_xmit_nit() in the trace is the first one called in
dev_hard_start_xmit() after the call to tcp_gso_segment(), i.e. it is
provided with the packet just processed by skb_segment().
[ 144.482460] tcp_skb_pcount 17, rounds 7, mss 1448, pre_len 24648 post_len 24616
[ 144.491462] tcp_skb_pcount 14, rounds 1, mss 1448, pre_len 20304 post_len 20272
[ 144.501836] Unable to handle kernel paging request at virtual address efe8f259
[ 144.509079] pgd = def84000
[ 144.511790] [efe8f259] *pgd=00000000
[ 144.515384] Internal error: Oops: 15 [#1] ARM
[ 144.519747] Modules linked in:
[ 144.522819] CPU: 0 PID: 3521 Comm: nginx Not tainted 3.13.0.rn102-00594-ga0fa1dd3cdbc-dirty #73
[ 144.531531] task: dd436700 ti: dfa5a000 task.ti: dfa5a000
[ 144.536947] PC is at kmem_cache_alloc+0x3c/0xe8
[ 144.541487] LR is at skb_clone+0x58/0xcc
[ 144.545417] pc : [<c0087ed8>] lr : [<c044829c>] psr: a00f0113
[ 144.545417] sp : dfa5ba40 ip : 0b50a8c0 fp : c05a2474
[ 144.556913] r10: dfa84540 r9 : 00000004 r8 : 00e2879e
[ 144.562146] r7 : c044829c r6 : 00000020 r5 : efe8f259 r4 : df801e00
[ 144.568683] r3 : 00000000 r2 : 00000000 r1 : 00000020 r0 : df801e00
[ 144.575221] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 144.582368] Control: 10c5387d Table: 1ef84019 DAC: 00000015
[ 144.588122] Process nginx (pid: 3521, stack limit = 0xdfa5a238)
[ 144.594051] Stack: (0xdfa5ba40 to 0xdfa5c000)
[ 144.598419] ba40: dfa84540 00000020 c0774620 df1ff800 00000000 c044829c ded01c00 ded01f0c
[ 144.606612] ba60: 00000000 c0451c94 df38a074 df1ff800 c05a2474 c0454ff0 00000004 ded01f20
[ 144.614806] ba80: c0774618 c0774620 dfa84540 c077c368 dfb7e530 df38a074 df1ff800 c0455364
[ 144.622999] baa0: 00000001 c0471e64 c0494634 df1ff800 00000000 00000000 00000000 ded57700
[ 144.631193] bac0: df1ff800 dfb7e530 df1ff800 df38a074 00000010 c046bd18 ded57700 00000000
[ 144.639386] bae0: 00000042 00000000 df38a074 df1ff800 dfb7e530 c04557dc ded57760 0002018a
[ 144.647579] bb00: 00000000 dd44cb40 dd44cbbc 0000000e 00000000 dfb7e530 df1ff800 df0a6e00
[ 144.655773] bb20: 00000010 c0494828 80000000 0b50a8c0 00000000 dfb7e530 cf781400 00000000
[ 144.663966] bb40: cf781604 df0a6e00 00000000 009c0000 00000000 c0494c50 dfb7e530 c0494d80
[ 144.672160] bb60: dfb7e480 cf781400 dfb7e480 00000020 00000000 ffffc33b 00000020 cf781400
[ 144.680353] bb80: dfb7e530 c077c368 00000000 df0a6e00 00000000 c04a990c efe8f259 134d4ee2
[ 144.688546] bba0: df801780 00000000 00000002 00000000 00000000 ffffc33b 001995f3 00000000
[ 144.696740] bbc0: cf781400 cf781400 dfb7e480 000005a8 df1bc480 00004988 00000000 2c79279e
[ 144.704933] bbe0: 00000000 c04a9f8c 00000000 00000000 0000000d 0000000c 0000000c 00000018
[ 144.713127] bc00: cf781400 cf7814bc 00000002 d08210e4 d08210d0 c04a3f20 dfa84540 cf781400
[ 144.721320] bc20: d08210e4 dfa84540 cf781400 d08210d0 dfa84540 000033b8 c0774618 c04aa8fc
[ 144.729514] bc40: 00000020 cf781400 cf781400 c04a6988 00000000 00000014 00000001 00000002
[ 144.737707] bc60: dfa84540 cf781400 df0a6480 cf781400 d08210d0 dfa84540 000033b8 c04ae840
[ 144.745901] bc80: c048fc40 c0471e64 c048fc40 c0471e64 dec1b300 c0774b64 dfa84540 dfa84540
[ 144.754095] bca0: c07a7e70 00000000 cf781400 c04b1004 df1ff800 c0471ee4 00000000 dfa5bcd4
[ 144.762288] bcc0: c048fc40 80000000 c048f958 c0774b64 00000010 c0774b64 3050a8c0 c05accb4
[ 144.770482] bce0: c0777470 00000000 c07a7e70 dfa84540 dfa84540 00000000 c0774618 c048fcd4
[ 144.778675] bd00: d08210d0 c07779e8 c0774628 df1ff800 dfa84540 c048fa80 ded01c00 c0774614
[ 144.786869] bd20: c0774614 c07779e8 c0774628 df1ff800 00000008 c045123c 00000003 00000000
[ 144.795062] bd40: e1448080 42000000 df1ff800 dfa84540 00000001 c0774628 00000000 dfa84540
[ 144.803256] bd60: 00000003 df3bd400 e1448080 42000000 df1ff800 dfa84540 00000001 c0453760
[ 144.811449] bd80: 00000004 c00194b0 e1448000 c03841c0 df5aca80 00020040 dedfa248 dee89c00
[ 144.819642] bda0: 00000001 df1ffbe0 00000001 0000004e 00000000 00000000 00001000 00000040
[ 144.827836] bdc0: df1ffbe0 00000100 00000000 00000100 df3bd400 c079f71c 00000000 c03842dc
[ 144.836029] bde0: c0dbe240 c00ad218 00020040 df1ff800 00000100 df1ffbfc 000038d0 00000000
[ 144.844223] be00: dfa5be48 c00ad2b8 00000000 dfa5be48 dfb4c600 c0384238 df1ffbfc 00000040
[ 144.852416] be20: 0000012c c07afbc0 c07afbc8 c077c368 c07afbc0 c04534ac 00000000 ffffc33d
[ 144.860610] be40: dfb4c600 00000001 0000000c c07b0950 c07b0940 dfa5a000 00000003 00000100
[ 144.868803] be60: 0000000c c001ec54 52e6c1c4 1ee3afbb de8a4760 0000000a ffffc33c 00404140
[ 144.876997] be80: 000003ff c078ef20 00000018 00000000 000003ff c07fb040 c0774048 0000ffff
[ 144.885190] bea0: 00000000 c001ef98 c078ef20 c000eac4 c008bf04 c07fb040 dfa5bee8 c00084dc
[ 144.893384] bec0: c008bee8 c008bf04 400f0013 ffffffff dfa5bf1c 00000000 dd423780 dfa5bf88
[ 144.901577] bee0: 00000000 c0011140 df5acaa0 00000002 dd423788 00000000 df5acaa0 00000002
[ 144.909771] bf00: dd423000 00032e80 00000000 dd423780 dfa5bf88 00000000 dd423788 dfa5bf30
[ 144.917965] bf20: c008bee8 c008bf04 400f0013 ffffffff 00000000 00000000 df5acaa0 de8a4760
[ 144.926158] bf40: dd423788 00000000 0066d8d0 00000000 00000000 00000000 00000000 00000000
[ 144.934351] bf60: 00000000 bed61258 dfa5a000 0000000a dfa5a000 00000000 00000000 c008cca8
[ 144.942545] bf80: ffffffff 000007ff 0063aa50 00000000 bed61258 00000000 00a29cf0 000000ef
[ 144.950738] bfa0: c000e384 c000e200 bed61258 00000000 00000008 0000000a bed61258 3f9c55b0
[ 144.958932] bfc0: bed61258 00000000 00a29cf0 000000ef 7fffefff 00000000 3f9c55b0 00000000
[ 144.967125] bfe0: 0008b3b0 bed611cc 000251a3 b6b5298c 000f0010 00000008 00000000 00000000
[ 144.975325] [<c0087ed8>] (kmem_cache_alloc+0x3c/0xe8) from [<c044829c>] (skb_clone+0x58/0xcc)
[ 144.983875] [<c044829c>] (skb_clone+0x58/0xcc) from [<c0451c94>] (dev_queue_xmit_nit+0x114/0x214)
[ 144.992768] [<c0451c94>] (dev_queue_xmit_nit+0x114/0x214) from [<c0455364>] (dev_hard_start_xmit+0x1c8/0x484)
[ 145.002711] [<c0455364>] (dev_hard_start_xmit+0x1c8/0x484) from [<c046bd18>] (sch_direct_xmit+0xa4/0x19c)
[ 145.012298] [<c046bd18>] (sch_direct_xmit+0xa4/0x19c) from [<c04557dc>] (__dev_queue_xmit+0x1bc/0x3dc)
[ 145.021629] [<c04557dc>] (__dev_queue_xmit+0x1bc/0x3dc) from [<c0494828>] (ip_finish_output+0x1f4/0x440)
[ 145.031129] [<c0494828>] (ip_finish_output+0x1f4/0x440) from [<c0494c50>] (ip_local_out+0x28/0x2c)
[ 145.040107] [<c0494c50>] (ip_local_out+0x28/0x2c) from [<c0494d80>] (ip_queue_xmit+0x12c/0x384)
[ 145.048828] [<c0494d80>] (ip_queue_xmit+0x12c/0x384) from [<c04a990c>] (tcp_transmit_skb+0x42c/0x86c)
[ 145.058067] [<c04a990c>] (tcp_transmit_skb+0x42c/0x86c) from [<c04a9f8c>] (tcp_write_xmit+0x174/0xa74)
[ 145.067393] [<c04a9f8c>] (tcp_write_xmit+0x174/0xa74) from [<c04aa8fc>] (__tcp_push_pending_frames+0x30/0x98)
[ 145.077328] [<c04aa8fc>] (__tcp_push_pending_frames+0x30/0x98) from [<c04a6988>] (tcp_rcv_established+0x144/0x5a0)
[ 145.087699] [<c04a6988>] (tcp_rcv_established+0x144/0x5a0) from [<c04ae840>] (tcp_v4_do_rcv+0x104/0x240)
[ 145.097200] [<c04ae840>] (tcp_v4_do_rcv+0x104/0x240) from [<c04b1004>] (tcp_v4_rcv+0x6ec/0x728)
[ 145.105918] [<c04b1004>] (tcp_v4_rcv+0x6ec/0x728) from [<c048fcd4>] (ip_local_deliver_finish+0x94/0x21c)
[ 145.115417] [<c048fcd4>] (ip_local_deliver_finish+0x94/0x21c) from [<c048fa80>] (ip_rcv_finish+0x128/0x2e8)
[ 145.125178] [<c048fa80>] (ip_rcv_finish+0x128/0x2e8) from [<c045123c>] (__netif_receive_skb_core+0x4c4/0x5d0)
[ 145.135115] [<c045123c>] (__netif_receive_skb_core+0x4c4/0x5d0) from [<c0453760>] (napi_gro_receive+0x74/0xa0)
[ 145.145139] [<c0453760>] (napi_gro_receive+0x74/0xa0) from [<c03841c0>] (mvneta_rx+0x420/0x498)
[ 145.153856] [<c03841c0>] (mvneta_rx+0x420/0x498) from [<c03842dc>] (mvneta_poll+0xa4/0x3b8)
[ 145.162225] [<c03842dc>] (mvneta_poll+0xa4/0x3b8) from [<c04534ac>] (net_rx_action+0x98/0x180)
[ 145.170858] [<c04534ac>] (net_rx_action+0x98/0x180) from [<c001ec54>] (__do_softirq+0xc8/0x1f4)
[ 145.179575] [<c001ec54>] (__do_softirq+0xc8/0x1f4) from [<c001ef98>] (irq_exit+0x6c/0xa8)
[ 145.187774] [<c001ef98>] (irq_exit+0x6c/0xa8) from [<c000eac4>] (handle_IRQ+0x34/0x84)
[ 145.195709] [<c000eac4>] (handle_IRQ+0x34/0x84) from [<c00084dc>] (armada_370_xp_handle_irq+0x4c/0xbc)
[ 145.205038] [<c00084dc>] (armada_370_xp_handle_irq+0x4c/0xbc) from [<c0011140>] (__irq_svc+0x40/0x50)
[ 145.214271] Exception stack(0xdfa5bee8 to 0xdfa5bf30)
[ 145.219333] bee0: df5acaa0 00000002 dd423788 00000000 df5acaa0 00000002
[ 145.227526] bf00: dd423000 00032e80 00000000 dd423780 dfa5bf88 00000000 dd423788 dfa5bf30
[ 145.235717] bf20: c008bee8 c008bf04 400f0013 ffffffff
[ 145.240786] [<c0011140>] (__irq_svc+0x40/0x50) from [<c008bf04>] (do_sendfile+0x270/0x314)
[ 145.249070] [<c008bf04>] (do_sendfile+0x270/0x314) from [<c008cca8>] (SyS_sendfile64+0x94/0xd0)
[ 145.257787] [<c008cca8>] (SyS_sendfile64+0x94/0xd0) from [<c000e200>] (ret_fast_syscall+0x0/0x30)
[ 145.266677] Code: e5935000 e3550000 0a00001b e5943014 (e7950003)
[ 145.272781] ---[ end trace 26c398308530643d ]---
[ 145.277405] Kernel panic - not syncing: Fatal exception in interrupt
[ 145.283818] Unable to handle kernel paging request at virtual address 2c78f4b6
[ 145.291050] pgd = def84000
[ 145.293761] [2c78f4b6] *pgd=00000000
[ 145.297351] Internal error: Oops: 15 [#2] ARM
[ 145.301713] Modules linked in:
[ 145.304784] CPU: 0 PID: 3521 Comm: nginx Tainted: G D 3.13.0.rn102-00594-ga0fa1dd3cdbc-dirty #73
[ 145.314452] task: dd436700 ti: dfa5a000 task.ti: dfa5a000
[ 145.319868] PC is at ata_scsi_qc_complete+0x24/0x360
[ 145.324842] LR is at __ata_qc_complete+0x94/0x13c
[ 145.329554] pc : [<c0345c9c>] lr : [<c033fc78>] psr: 400f0193
[ 145.329554] sp : dfa5b6d0 ip : 00000001 fp : 00000001
[ 145.341049] r10: df347910 r9 : 00000001 r8 : 00000001
[ 145.346281] r7 : df390000 r6 : dfa84900 r5 : 00000000 r4 : df3900d0
[ 145.352819] r3 : c0345c78 r2 : 2c78f4b6 r1 : 00000003 r0 : df3900d0
[ 145.359357] Flags: nZcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[ 145.366590] Control: 10c5387d Table: 1ef84019 DAC: 00000015
[ 145.372344] Process nginx (pid: 3521, stack limit = 0xdfa5a238)
[ 145.378272] Stack: (0xdfa5b6d0 to 0xdfa5c000)
[ 145.382637] b6c0: df347910 c033f9d8 00000000 c066840b
[ 145.390831] b6e0: c066840b df3900d0 df390000 df391440 00000000 00000001 df347910 c033fc78
[ 145.399024] b700: 00000000 00000001 df390000 c033ffd0 00000000 c0349f8c df390000 00000008
[ 145.407217] b720: e08d4100 00000001 df315410 c0352808 c030f660 c07fcd0c 0000000a c03094a8
[ 145.415410] b740: c07b2ef4 c07b2eac 00000048 00000000 df347a90 00000001 df347a90 e08d4000
[ 145.423603] b760: 00000001 00000001 00000001 c0353988 df36fa80 00000069 00000000 00000000
[ 145.431797] b780: 00000069 c07afc83 df347a00 c0042480 c07b29d8 c003f8e8 00000001 df347a00
[ 145.439990] b7a0: 00000069 dfa5bee8 000003ff c07fb040 c0774048 0000ffff 00010000 c004260c
[ 145.448183] b7c0: 00000069 c00443f8 00000069 dfa5bee8 00000069 c0041df8 c078ef20 c000eac0
[ 145.456377] b7e0: 00000010 c07fb040 dfa5b818 c0008540 c0040394 c0558b98 600f0113 ffffffff
[ 145.464570] b800: dfa5b84c 00000000 dfa5a000 c07b04a8 c07b0304 c0011140 0000019b 00000000
[ 145.472763] b820: c07b32cc 00000000 c07b04a8 c07b04a8 dfa5a000 00000000 00000000 dfa5a000
[ 145.480957] b840: c07b04a8 c07b0304 600f0193 dfa5b860 c0040394 c0558b98 600f0113 ffffffff
[ 145.489150] b860: c07b0304 dfa5b88c 0000000b dfa5b8e6 dfa5a000 c077400c c077a5e8 dfa5a000
[ 145.497343] b880: 00000001 c00108b8 c0664a04 00000000 00000500 00000500 dfa5a238 0000000b
[ 145.505537] b8a0: 600f0193 00000000 00000008 bf000000 00000000 65000000 35333935 20303030
[ 145.513730] b8c0: 35353365 30303030 30613020 31303030 35652062 30333439 28203431 35393765
[ 145.521923] b8e0: 33303030 df002029 c05a2474 c2648667 c06b1808 efe8f259 dee99180 00000015
[ 145.530117] b900: dfa5b9f8 dfa5b9f8 00000004 dfa84540 c05a2474 c0558a20 dee99180 c0014518
[ 145.538310] b920: df38b808 00000020 df801e00 00000015 c0004000 00000000 def84000 c0014580
[ 145.546503] b940: 00000005 00000015 c0014518 c077a81c efe8f259 c0008394 df25d800 c044adb4
[ 145.554696] b960: 00000003 dee546b8 00001e72 c07a7e70 00002d82 0000332a dfb7e530 00000000
[ 145.562889] b980: 000005a8 00000042 0000008e 00000000 00000001 dfa84540 000000d0 000005a8
[ 145.571084] b9a0: 00000000 00000042 00000001 ffffffbe dfa84540 00010000 00000000 c04a861c
[ 145.579277] b9c0: 00000000 dfb7e530 00005b27 df38b800 dee1aad0 c04b42a8 dee54654 000061dc
[ 145.587470] b9e0: c0087ed8 a00f0113 ffffffff dfa5ba2c 00e2879e c00110d8 df801e00 00000020
[ 145.595663] ba00: 00000000 00000000 df801e00 efe8f259 00000020 c044829c 00e2879e 00000004
[ 145.603857] ba20: dfa84540 c05a2474 0b50a8c0 dfa5ba40 c044829c c0087ed8 a00f0113 ffffffff
[ 145.612050] ba40: dfa84540 00000020 c0774620 df1ff800 00000000 c044829c ded01c00 ded01f0c
[ 145.620243] ba60: 00000000 c0451c94 df38a074 df1ff800 c05a2474 c0454ff0 00000004 ded01f20
[ 145.628437] ba80: c0774618 c0774620 dfa84540 c077c368 dfb7e530 df38a074 df1ff800 c0455364
[ 145.636630] baa0: 00000001 c0471e64 c0494634 df1ff800 00000000 00000000 00000000 ded57700
[ 145.644823] bac0: df1ff800 dfb7e530 df1ff800 df38a074 00000010 c046bd18 ded57700 00000000
[ 145.653017] bae0: 00000042 00000000 df38a074 df1ff800 dfb7e530 c04557dc ded57760 0002018a
[ 145.661210] bb00: 00000000 dd44cb40 dd44cbbc 0000000e 00000000 dfb7e530 df1ff800 df0a6e00
[ 145.669403] bb20: 00000010 c0494828 80000000 0b50a8c0 00000000 dfb7e530 cf781400 00000000
[ 145.677596] bb40: cf781604 df0a6e00 00000000 009c0000 00000000 c0494c50 dfb7e530 c0494d80
[ 145.685789] bb60: dfb7e480 cf781400 dfb7e480 00000020 00000000 ffffc33b 00000020 cf781400
[ 145.693983] bb80: dfb7e530 c077c368 00000000 df0a6e00 00000000 c04a990c efe8f259 134d4ee2
[ 145.702176] bba0: df801780 00000000 00000002 00000000 00000000 ffffc33b 001995f3 00000000
[ 145.710369] bbc0: cf781400 cf781400 dfb7e480 000005a8 df1bc480 00004988 00000000 2c79279e
[ 145.718562] bbe0: 00000000 c04a9f8c 00000000 00000000 0000000d 0000000c 0000000c 00000018
[ 145.726755] bc00: cf781400 cf7814bc 00000002 d08210e4 d08210d0 c04a3f20 dfa84540 cf781400
[ 145.734949] bc20: d08210e4 dfa84540 cf781400 d08210d0 dfa84540 000033b8 c0774618 c04aa8fc
[ 145.743143] bc40: 00000020 cf781400 cf781400 c04a6988 00000000 00000014 00000001 00000002
[ 145.751336] bc60: dfa84540 cf781400 df0a6480 cf781400 d08210d0 dfa84540 000033b8 c04ae840
[ 145.759530] bc80: c048fc40 c0471e64 c048fc40 c0471e64 dec1b300 c0774b64 dfa84540 dfa84540
[ 145.767724] bca0: c07a7e70 00000000 cf781400 c04b1004 df1ff800 c0471ee4 00000000 dfa5bcd4
[ 145.775918] bcc0: c048fc40 80000000 c048f958 c0774b64 00000010 c0774b64 3050a8c0 c05accb4
[ 145.784111] bce0: c0777470 00000000 c07a7e70 dfa84540 dfa84540 00000000 c0774618 c048fcd4
[ 145.792305] bd00: d08210d0 c07779e8 c0774628 df1ff800 dfa84540 c048fa80 ded01c00 c0774614
[ 145.800498] bd20: c0774614 c07779e8 c0774628 df1ff800 00000008 c045123c 00000003 00000000
[ 145.808691] bd40: e1448080 42000000 df1ff800 dfa84540 00000001 c0774628 00000000 dfa84540
[ 145.816884] bd60: 00000003 df3bd400 e1448080 42000000 df1ff800 dfa84540 00000001 c0453760
[ 145.825078] bd80: 00000004 c00194b0 e1448000 c03841c0 df5aca80 00020040 dedfa248 dee89c00
[ 145.833271] bda0: 00000001 df1ffbe0 00000001 0000004e 00000000 00000000 00001000 00000040
[ 145.841464] bdc0: df1ffbe0 00000100 00000000 00000100 df3bd400 c079f71c 00000000 c03842dc
[ 145.849658] bde0: c0dbe240 c00ad218 00020040 df1ff800 00000100 df1ffbfc 000038d0 00000000
[ 145.857852] be00: dfa5be48 c00ad2b8 00000000 dfa5be48 dfb4c600 c0384238 df1ffbfc 00000040
[ 145.866045] be20: 0000012c c07afbc0 c07afbc8 c077c368 c07afbc0 c04534ac 00000000 ffffc33d
[ 145.874239] be40: dfb4c600 00000001 0000000c c07b0950 c07b0940 dfa5a000 00000003 00000100
[ 145.882432] be60: 0000000c c001ec54 52e6c1c4 1ee3afbb de8a4760 0000000a ffffc33c 00404140
[ 145.890625] be80: 000003ff c078ef20 00000018 00000000 000003ff c07fb040 c0774048 0000ffff
[ 145.898819] bea0: 00000000 c001ef98 c078ef20 c000eac4 c008bf04 c07fb040 dfa5bee8 c00084dc
[ 145.907013] bec0: c008bee8 c008bf04 400f0013 ffffffff dfa5bf1c 00000000 dd423780 dfa5bf88
[ 145.915206] bee0: 00000000 c0011140 df5acaa0 00000002 dd423788 00000000 df5acaa0 00000002
[ 145.923401] bf00: dd423000 00032e80 00000000 dd423780 dfa5bf88 00000000 dd423788 dfa5bf30
[ 145.931594] bf20: c008bee8 c008bf04 400f0013 ffffffff 00000000 00000000 df5acaa0 de8a4760
[ 145.939787] bf40: dd423788 00000000 0066d8d0 00000000 00000000 00000000 00000000 00000000
[ 145.947980] bf60: 00000000 bed61258 dfa5a000 0000000a dfa5a000 00000000 00000000 c008cca8
[ 145.956174] bf80: ffffffff 000007ff 0063aa50 00000000 bed61258 00000000 00a29cf0 000000ef
[ 145.964367] bfa0: c000e384 c000e200 bed61258 00000000 00000008 0000000a bed61258 3f9c55b0
[ 145.972561] bfc0: bed61258 00000000 00a29cf0 000000ef 7fffefff 00000000 3f9c55b0 00000000
[ 145.980754] bfe0: 0008b3b0 bed611cc 000251a3 b6b5298c 000f0010 00000008 00000000 00000000
[ 145.988952] [<c0345c9c>] (ata_scsi_qc_complete+0x24/0x360) from [<c033fc78>] (__ata_qc_complete+0x94/0x13c)
[ 145.998714] [<c033fc78>] (__ata_qc_complete+0x94/0x13c) from [<c033ffd0>] (ata_qc_complete_multiple+0x98/0xd0)
[ 146.008741] [<c033ffd0>] (ata_qc_complete_multiple+0x98/0xd0) from [<c0352808>] (ahci_handle_port_interrupt+0x120/0x5b8)
[ 146.019634] [<c0352808>] (ahci_handle_port_interrupt+0x120/0x5b8) from [<c0353988>] (ahci_interrupt+0x60/0xe0)
[ 146.029663] [<c0353988>] (ahci_interrupt+0x60/0xe0) from [<c0042480>] (handle_irq_event_percpu+0x50/0x1b4)
[ 146.039338] [<c0042480>] (handle_irq_event_percpu+0x50/0x1b4) from [<c004260c>] (handle_irq_event+0x28/0x38)
[ 146.049186] [<c004260c>] (handle_irq_event+0x28/0x38) from [<c00443f8>] (handle_simple_irq+0x60/0x98)
[ 146.058426] [<c00443f8>] (handle_simple_irq+0x60/0x98) from [<c0041df8>] (generic_handle_irq+0x20/0x30)
[ 146.067839] [<c0041df8>] (generic_handle_irq+0x20/0x30) from [<c000eac0>] (handle_IRQ+0x30/0x84)
[ 146.076642] [<c000eac0>] (handle_IRQ+0x30/0x84) from [<c0008540>] (armada_370_xp_handle_irq+0xb0/0xbc)
[ 146.085969] [<c0008540>] (armada_370_xp_handle_irq+0xb0/0xbc) from [<c0011140>] (__irq_svc+0x40/0x50)
[ 146.095203] Exception stack(0xdfa5b818 to 0xdfa5b860)
[ 146.100262] b800: 0000019b 00000000
[ 146.108456] b820: c07b32cc 00000000 c07b04a8 c07b04a8 dfa5a000 00000000 00000000 dfa5a000
[ 146.116649] b840: c07b04a8 c07b0304 600f0193 dfa5b860 c0040394 c0558b98 600f0113 ffffffff
[ 146.124846] [<c0011140>] (__irq_svc+0x40/0x50) from [<c0558b98>] (panic+0x158/0x1c8)
[ 146.132606] [<c0558b98>] (panic+0x158/0x1c8) from [<c00108b8>] (die+0x194/0x350)
[ 146.140019] [<c00108b8>] (die+0x194/0x350) from [<c0558a20>] (__do_kernel_fault.part.10+0x54/0x74)
[ 146.148999] [<c0558a20>] (__do_kernel_fault.part.10+0x54/0x74) from [<c0014518>] (do_translation_fault+0x0/0xa0)
[ 146.159192] [<c0014518>] (do_translation_fault+0x0/0xa0) from [<00000000>] ( (null))
[ 146.167038] Code: e5907000 e5962030 e2955000 13a05001 (e5d23000)
[ 146.173140] ---[ end trace 26c398308530643e ]---
[ 146.177764] Kernel panic - not syncing: Fatal exception in interrupt
Cheers,
a+
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-01-27 22:15 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-01-22 21:46 [BUG] null pointer dereference in tcp_gso_segment() Arnaud Ebalard
2014-01-22 21:57 ` Eric Dumazet
2014-01-22 22:02 ` Arnaud Ebalard
2014-01-22 22:18 ` Eric Dumazet
2014-01-22 23:56 ` Willy Tarreau
2014-01-26 0:04 ` Arnaud Ebalard
2014-01-25 23:54 ` Arnaud Ebalard
2014-01-26 1:18 ` Eric Dumazet
2014-01-27 22:14 ` Arnaud Ebalard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).