From: Guenter Roeck <linux@roeck-us.net>
To: Eric Dumazet <edumazet@google.com>
Cc: Eric Dumazet <eric.dumazet@gmail.com>,
"David S . Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>, netdev <netdev@vger.kernel.org>,
Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
Jason Wang <jasowang@redhat.com>,
virtualization@lists.linux-foundation.org
Subject: Re: [PATCH net] virtio_net: Do not pull payload in skb->head
Date: Sun, 11 Apr 2021 14:32:50 -0700 [thread overview]
Message-ID: <c1d15bd0-8b62-f7c0-0f2e-1d527827f451@roeck-us.net> (raw)
In-Reply-To: <CANn89iJa8KAnfWvUB8Jr8hsG5x_Amg90DbpoAHiuNZigv75MEA@mail.gmail.com>
On 4/11/21 2:23 PM, Eric Dumazet wrote:
> On Sun, Apr 11, 2021 at 10:37 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> On 4/11/21 8:06 AM, Eric Dumazet wrote:
>>> On Sun, Apr 11, 2021 at 3:43 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>>
>>>> This patch causes a virtio-net interface failure when booting sh4 images
>>>> in qemu. The test case is nothing special: Just try to get an IP address
>>>> using udhcpc. If it fails, udhcpc reports:
>>>>
>>>> udhcpc: started, v1.33.0
>>>> udhcpc: sending discover
>>>> FAIL
>>>>
>>>
>>> Can you investigate where the incoming packet is dropped ?
>>>
>>
>> Unless I am missing something, packets are not dropped. It looks more
>> like udhcpc gets bad indigestion in the receive path and exits immediately.
>> Plus, it doesn't happen all the time; sometimes it receives the discover
>> response and is able to obtain an IP address.
>>
>> Overall this is quite puzzling since udhcpc exits immediately when the problem
>> is seen, no matter which option I give it on the command line; it should not
>> really do that.
>
>
> Could you strace both cases and report differences you can spot ?
>
> strace -o STRACE -f -s 1000 udhcpc
>
I'll give it a try. It will take a while; I'll need to add strace to my root
file systems first.
As a quick hack, I added some debugging into the kernel; it looks like
the data part of the dhcp discover response may get lost with your patch
in place.
dhcp discover response with patch in place (bad):
virtio_net virtio0 eth0: __udp4_lib_rcv: data 0x8ca4cc44 head 0x8ca4cc00 tail 0x8ca4cc4c len 556 datalen 548 caller ip_protocol_deliver_rcu+0xac/0x178
00000000: 70 c1 a9 8c 00 00 00 00 00 00 00 00 20 ee c3 7b 34 00 e0 7b 08 00 00 00 00 00 00 00 00 00 00 00 p........... ..{4..{............
00000020: 60 c8 ff ff ff ff ff ff 52 55 0a 00 02 02 08 00 45 10 02 40 00 00 00 00 40 11 6c 9c 0a 00 02 02 `.......RU......E..@....@.l.....
00000040: ff ff ff ff 00 43 00 44 02 2c e1 21 00 00 00 00 f0 6f a4 7b 00 00 80 00 ff ff ff ff 7f 45 4c 46 .....C.D.,.!.....o.{.........ELF
^^ udp header
^^^^^ UDP length (556)
^^ start of UDP data (dhcp discover reply)
00000060: 01 01 01 00 00 00 00 00 00 00 00 00 02 00 2a 00 01 00 00 00 b0 6e 40 00 34 00 00 00 2c f6 0a 00 ..............*......n@.4...,...
00000080: 17 00 00 00 34 00 20 00 08 00 28 00 16 00 15 00 06 00 00 00 34 00 00 00 34 00 40 00 34 00 40 00 ....4. ...(.........4...4.@.4.@.
000000a0: 00 01 00 00 00 01 00 00 04 00 00 00 04 00 00 00 03 00 00 00 34 01 00 00 34 01 40 00 34 01 40 00 ....................4...4.@.4.@.
000000c0: 15 00 00 00 15 00 00 00 04 00 00 00 01 00 00 00 01 00 00 00 00 00 00 00 00 00 40 00 00 00 40 00 ..........................@...@.
000000e0: 1c e2 0a 00 1c e2 0a 00 05 00 00 00 00 00 01 00 01 00 00 00 38 ef 0a 00 38 ef 4b 00 38 ef 4b 00 ....................8...8.K.8.K.
00000100: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................................
00000120: b8 00 00 00 00 4c f9 8f 24 02 00 00 36 00 00 00 50 e5 74 64 88 e0 0a 00 88 e0 4a 00 88 e0 4a 00 .....L..$...6...P.td......J...J.
00000140: 2c 00 00 00 2c 00 00 00 04 00 00 00 04 00 00 00 51 e5 74 64 00 00 00 00 00 00 00 00 00 00 00 00 ,...,...........Q.td............
00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000200: 1c 63 5a 8c 00 3e a4 8c 88 f9 50 8c 0c ce a4 8c 0c ce a4 8c 24 81 a1 8c 00 00 00 00 1c 5c 5a 8c .cZ..>....P.........$........\Z.
00000220: 04 b8 a5 8c 01 00 00 00 03 00 00 00 ............
dhcp discover response with patch reverted (ok):
virtio_net virtio0 eth0: __udp4_lib_rcv: data 0x8ca4ca44 head 0x8ca4ca00 tail 0x8ca4cb00 len 556 datalen 368 caller ip_protocol_deliver_rcu+0xac/0x178
^^^^^^^^^^ ^^^^^^^^^^ ^^^
00000000: 4c bd ab 8c 00 00 00 00 00 00 00 00 20 2e 85 7b 34 00 e0 7b 08 00 00 00 00 00 00 00 00 00 00 00 L........... ..{4..{............
00000020: 40 9a ff ff ff ff ff ff 52 55 0a 00 02 02 08 00 45 10 02 40 00 00 00 00 40 11 6c 9c 0a 00 02 02 @.......RU......E..@....@.l.....
^^^^^ ip length (576)
00000040: ff ff ff ff 00 43 00 44 02 2c 06 18 02 01 06 00 e6 fd ce 5b 00 00 00 00 00 00 00 00 0a 00 02 0f .....C.D.,.........[............
^^ udp header
^^^^^ UDP length (556)
^^ start of UDP data
00000060: 0a 00 02 02 00 00 00 00 52 54 00 12 34 56 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........RT..4V..................
00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000100: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 ................................
00000120: b8 00 00 00 00 4d f9 8f 70 01 00 00 ea 00 00 00 50 e5 74 64 88 e0 0a 00 88 e0 4a 00 88 e0 4a 00 .....M..p.......P.td......J...J.
00000140: 2c 00 00 00 2c 00 00 00 04 00 00 00 04 00 00 00 51 e5 74 64 00 00 00 00 00 00 00 00 00 00 00 00 ,...,...........Q.td............
00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000200: 00 00 00 00 00 00 00 00 06 00 01 00 54 cc a4 8c 08 00 00 00 02 00 00 00 00 00 00 00 01 00 00 00 ............T...................
00000220: 08 00 00 00 00 00 00 00 14 cd a4 8c ............
WARNING: multiple messages have this Message-ID (diff)
From: Guenter Roeck <linux@roeck-us.net>
To: Eric Dumazet <edumazet@google.com>
Cc: Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
Eric Dumazet <eric.dumazet@gmail.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
netdev <netdev@vger.kernel.org>,
virtualization@lists.linux-foundation.org,
Jakub Kicinski <kuba@kernel.org>,
"David S . Miller" <davem@davemloft.net>
Subject: Re: [PATCH net] virtio_net: Do not pull payload in skb->head
Date: Sun, 11 Apr 2021 14:32:50 -0700 [thread overview]
Message-ID: <c1d15bd0-8b62-f7c0-0f2e-1d527827f451@roeck-us.net> (raw)
In-Reply-To: <CANn89iJa8KAnfWvUB8Jr8hsG5x_Amg90DbpoAHiuNZigv75MEA@mail.gmail.com>
On 4/11/21 2:23 PM, Eric Dumazet wrote:
> On Sun, Apr 11, 2021 at 10:37 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> On 4/11/21 8:06 AM, Eric Dumazet wrote:
>>> On Sun, Apr 11, 2021 at 3:43 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>>
>>>> This patch causes a virtio-net interface failure when booting sh4 images
>>>> in qemu. The test case is nothing special: Just try to get an IP address
>>>> using udhcpc. If it fails, udhcpc reports:
>>>>
>>>> udhcpc: started, v1.33.0
>>>> udhcpc: sending discover
>>>> FAIL
>>>>
>>>
>>> Can you investigate where the incoming packet is dropped ?
>>>
>>
>> Unless I am missing something, packets are not dropped. It looks more
>> like udhcpc gets bad indigestion in the receive path and exits immediately.
>> Plus, it doesn't happen all the time; sometimes it receives the discover
>> response and is able to obtain an IP address.
>>
>> Overall this is quite puzzling since udhcpc exits immediately when the problem
>> is seen, no matter which option I give it on the command line; it should not
>> really do that.
>
>
> Could you strace both cases and report differences you can spot ?
>
> strace -o STRACE -f -s 1000 udhcpc
>
I'll give it a try. It will take a while; I'll need to add strace to my root
file systems first.
As a quick hack, I added some debugging into the kernel; it looks like
the data part of the dhcp discover response may get lost with your patch
in place.
dhcp discover response with patch in place (bad):
virtio_net virtio0 eth0: __udp4_lib_rcv: data 0x8ca4cc44 head 0x8ca4cc00 tail 0x8ca4cc4c len 556 datalen 548 caller ip_protocol_deliver_rcu+0xac/0x178
00000000: 70 c1 a9 8c 00 00 00 00 00 00 00 00 20 ee c3 7b 34 00 e0 7b 08 00 00 00 00 00 00 00 00 00 00 00 p........... ..{4..{............
00000020: 60 c8 ff ff ff ff ff ff 52 55 0a 00 02 02 08 00 45 10 02 40 00 00 00 00 40 11 6c 9c 0a 00 02 02 `.......RU......E..@....@.l.....
00000040: ff ff ff ff 00 43 00 44 02 2c e1 21 00 00 00 00 f0 6f a4 7b 00 00 80 00 ff ff ff ff 7f 45 4c 46 .....C.D.,.!.....o.{.........ELF
^^ udp header
^^^^^ UDP length (556)
^^ start of UDP data (dhcp discover reply)
00000060: 01 01 01 00 00 00 00 00 00 00 00 00 02 00 2a 00 01 00 00 00 b0 6e 40 00 34 00 00 00 2c f6 0a 00 ..............*......n@.4...,...
00000080: 17 00 00 00 34 00 20 00 08 00 28 00 16 00 15 00 06 00 00 00 34 00 00 00 34 00 40 00 34 00 40 00 ....4. ...(.........4...4.@.4.@.
000000a0: 00 01 00 00 00 01 00 00 04 00 00 00 04 00 00 00 03 00 00 00 34 01 00 00 34 01 40 00 34 01 40 00 ....................4...4.@.4.@.
000000c0: 15 00 00 00 15 00 00 00 04 00 00 00 01 00 00 00 01 00 00 00 00 00 00 00 00 00 40 00 00 00 40 00 ..........................@...@.
000000e0: 1c e2 0a 00 1c e2 0a 00 05 00 00 00 00 00 01 00 01 00 00 00 38 ef 0a 00 38 ef 4b 00 38 ef 4b 00 ....................8...8.K.8.K.
00000100: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................................
00000120: b8 00 00 00 00 4c f9 8f 24 02 00 00 36 00 00 00 50 e5 74 64 88 e0 0a 00 88 e0 4a 00 88 e0 4a 00 .....L..$...6...P.td......J...J.
00000140: 2c 00 00 00 2c 00 00 00 04 00 00 00 04 00 00 00 51 e5 74 64 00 00 00 00 00 00 00 00 00 00 00 00 ,...,...........Q.td............
00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000200: 1c 63 5a 8c 00 3e a4 8c 88 f9 50 8c 0c ce a4 8c 0c ce a4 8c 24 81 a1 8c 00 00 00 00 1c 5c 5a 8c .cZ..>....P.........$........\Z.
00000220: 04 b8 a5 8c 01 00 00 00 03 00 00 00 ............
dhcp discover response with patch reverted (ok):
virtio_net virtio0 eth0: __udp4_lib_rcv: data 0x8ca4ca44 head 0x8ca4ca00 tail 0x8ca4cb00 len 556 datalen 368 caller ip_protocol_deliver_rcu+0xac/0x178
^^^^^^^^^^ ^^^^^^^^^^ ^^^
00000000: 4c bd ab 8c 00 00 00 00 00 00 00 00 20 2e 85 7b 34 00 e0 7b 08 00 00 00 00 00 00 00 00 00 00 00 L........... ..{4..{............
00000020: 40 9a ff ff ff ff ff ff 52 55 0a 00 02 02 08 00 45 10 02 40 00 00 00 00 40 11 6c 9c 0a 00 02 02 @.......RU......E..@....@.l.....
^^^^^ ip length (576)
00000040: ff ff ff ff 00 43 00 44 02 2c 06 18 02 01 06 00 e6 fd ce 5b 00 00 00 00 00 00 00 00 0a 00 02 0f .....C.D.,.........[............
^^ udp header
^^^^^ UDP length (556)
^^ start of UDP data
00000060: 0a 00 02 02 00 00 00 00 52 54 00 12 34 56 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........RT..4V..................
00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000100: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 ................................
00000120: b8 00 00 00 00 4d f9 8f 70 01 00 00 ea 00 00 00 50 e5 74 64 88 e0 0a 00 88 e0 4a 00 88 e0 4a 00 .....M..p.......P.td......J...J.
00000140: 2c 00 00 00 2c 00 00 00 04 00 00 00 04 00 00 00 51 e5 74 64 00 00 00 00 00 00 00 00 00 00 00 00 ,...,...........Q.td............
00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000200: 00 00 00 00 00 00 00 00 06 00 01 00 54 cc a4 8c 08 00 00 00 02 00 00 00 00 00 00 00 01 00 00 00 ............T...................
00000220: 08 00 00 00 00 00 00 00 14 cd a4 8c ............
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2021-04-11 21:32 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-02 13:26 [PATCH net] virtio_net: Do not pull payload in skb->head Eric Dumazet
2021-04-02 13:26 ` Eric Dumazet
2021-04-06 2:03 ` Jason Wang
2021-04-06 2:03 ` Jason Wang
2021-04-07 18:09 ` Michael S. Tsirkin
2021-04-07 18:09 ` Michael S. Tsirkin
2021-04-11 13:43 ` Guenter Roeck
2021-04-11 13:43 ` Guenter Roeck
2021-04-11 15:06 ` Eric Dumazet
2021-04-11 20:37 ` Guenter Roeck
2021-04-11 20:37 ` Guenter Roeck
2021-04-11 21:23 ` Eric Dumazet
2021-04-11 21:32 ` Guenter Roeck [this message]
2021-04-11 21:32 ` Guenter Roeck
2021-04-11 21:43 ` Eric Dumazet
2021-04-11 22:07 ` Guenter Roeck
2021-04-11 22:07 ` Guenter Roeck
2021-04-12 5:48 ` Eric Dumazet
2021-04-12 5:53 ` Eric Dumazet
2021-04-12 6:23 ` Guenter Roeck
2021-04-12 6:23 ` Guenter Roeck
2021-04-12 6:37 ` Eric Dumazet
2021-04-11 22:20 ` Guenter Roeck
2021-04-11 22:20 ` Guenter Roeck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c1d15bd0-8b62-f7c0-0f2e-1d527827f451@roeck-us.net \
--to=linux@roeck-us.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=eric.dumazet@gmail.com \
--cc=jasowang@redhat.com \
--cc=kuba@kernel.org \
--cc=mst@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=virtualization@lists.linux-foundation.org \
--cc=xuanzhuo@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.