network performance degradation in virtio_net in 4.12-rc

network performance degradation in virtio_net in 4.12-rc

[Mikulas Patocka]

Hi

I see severe network performance degradation with the kernels 4.12-rc1 and 
4.12-rc2 in the network virtio driver. Download rate drops down to about 
100kB/s.

I bisected it and it is caused by patch 
d85b758f72b05a774045545f24d70980e3e9aac4 ("virtio_net: fix support for 
small rings"). When I revert this patch, the problem goes away.

The host is Debian Jessie with kernel 4.4.62, the guest is Debian Sid with 
kernel 4.12-rc.

Mikulas

Re: network performance degradation in virtio_net in 4.12-rc

[Michael S. Tsirkin]

On Mon, May 22, 2017 at 10:25:19AM -0400, Mikulas Patocka wrote:
> Hi
> 
> I see severe network performance degradation with the kernels 4.12-rc1 and 
> 4.12-rc2 in the network virtio driver. Download rate drops down to about 
> 100kB/s.
> 
> I bisected it and it is caused by patch 
> d85b758f72b05a774045545f24d70980e3e9aac4 ("virtio_net: fix support for 
> small rings"). When I revert this patch, the problem goes away.
> 
> The host is Debian Jessie with kernel 4.4.62, the guest is Debian Sid with 
> kernel 4.12-rc.
> 
> Mikulas

Thanks for the report!
I'm not sure what causes it on this host. I'll try to reproduce.
Meanwhile, could you please locate this line of code:
+               vi->rq[i].min_buf_len = mergeable_min_buf_len(vi, vi->rq[i].vq);

and add something like
	printk(KERN_ERR, "min buf = 0x%x expected 0x%x size 0x%x big %d\n",
	       vi->rq[i].min_buf_len, GOOD_PACKET_LEN,
	       virtqueue_get_vring_size(vi->rq[i].vq),
	       (int)vi->big_packets);

after it?
Then boot and capture the output.

Thanks!

-- 
MST

Re: network performance degradation in virtio_net in 4.12-rc

[Michael S. Tsirkin]

On Mon, May 22, 2017 at 10:25:19AM -0400, Mikulas Patocka wrote:
> Hi
> 
> I see severe network performance degradation with the kernels 4.12-rc1 and 
> 4.12-rc2 in the network virtio driver. Download rate drops down to about 
> 100kB/s.
> 
> I bisected it and it is caused by patch 
> d85b758f72b05a774045545f24d70980e3e9aac4 ("virtio_net: fix support for 
> small rings"). When I revert this patch, the problem goes away.
> 
> The host is Debian Jessie with kernel 4.4.62, the guest is Debian Sid with 
> kernel 4.12-rc.
> 
> Mikulas

Thanks for the report!
I'm not sure what causes it on this host. I'll try to reproduce.
Meanwhile, could you please locate this line of code:
+               vi->rq[i].min_buf_len = mergeable_min_buf_len(vi, vi->rq[i].vq);

and add something like
	printk(KERN_ERR, "min buf = 0x%x expected 0x%x size 0x%x big %d\n",
	       vi->rq[i].min_buf_len, GOOD_PACKET_LEN,
	       virtqueue_get_vring_size(vi->rq[i].vq),
	       (int)vi->big_packets);

after it?
Then boot and capture the output.

Thanks!

-- 
MST

Re: network performance degradation in virtio_net in 4.12-rc

[Mikulas Patocka]

On Mon, 22 May 2017, Michael S. Tsirkin wrote:

> On Mon, May 22, 2017 at 10:25:19AM -0400, Mikulas Patocka wrote:
> > Hi
> > 
> > I see severe network performance degradation with the kernels 4.12-rc1 and 
> > 4.12-rc2 in the network virtio driver. Download rate drops down to about 
> > 100kB/s.
> > 
> > I bisected it and it is caused by patch 
> > d85b758f72b05a774045545f24d70980e3e9aac4 ("virtio_net: fix support for 
> > small rings"). When I revert this patch, the problem goes away.
> > 
> > The host is Debian Jessie with kernel 4.4.62, the guest is Debian Sid with 
> > kernel 4.12-rc.
> > 
> > Mikulas
> 
> Thanks for the report!
> I'm not sure what causes it on this host. I'll try to reproduce.
> Meanwhile, could you please locate this line of code:
> +               vi->rq[i].min_buf_len = mergeable_min_buf_len(vi, vi->rq[i].vq);
> 
> and add something like
> 	printk(KERN_ERR, "min buf = 0x%x expected 0x%x size 0x%x big %d\n",
> 	       vi->rq[i].min_buf_len, GOOD_PACKET_LEN,
> 	       virtqueue_get_vring_size(vi->rq[i].vq),
> 	       (int)vi->big_packets);
> 
> after it?
> Then boot and capture the output.
> 
> Thanks!

It prints one line "min buf = 0x101 expected 0x5ee size 0x100 big 1".

Mikulas

network performance degradation in virtio_net in 4.12-rc

[Mikulas Patocka]

Hi

I see severe network performance degradation with the kernels 4.12-rc1 and 
4.12-rc2 in the network virtio driver. Download rate drops down to about 
100kB/s.

I bisected it and it is caused by patch 
d85b758f72b05a774045545f24d70980e3e9aac4 ("virtio_net: fix support for 
small rings"). When I revert this patch, the problem goes away.

The host is Debian Jessie with kernel 4.4.62, the guest is Debian Sid with 
kernel 4.12-rc.

Mikulas