Question about KVM IO performance with FreeBSD as a guest OS

Question about KVM IO performance with FreeBSD as a guest OS

[Rainer Duffner]

Hi,

I have huge problems running FreeBSD 12 (amd64) as a KVM guest.

KVM is running on Ubuntu 18 LTS, in an OpenStack setup with dedicated Ceph-Storage (NVMe SSDs).

The VM „flavor" as such is that IOPs are limited to 2000/s - and I do get those 2k IOPs when I run e.g. CentOS 7.

But on FreeBSD, I get way less.

E.g. running dc3dd to write zeros to a disk, I get 120 MB/s on CentOS 7.
With FreeBSD, I get 9 MB/s.


The VMs were created on an OpenSuSE 42.3 host with the commands described here:

https://docs.openstack.org/image-guide/freebsd-image.html


This mimics the results we got on XenServer, where also some people reported the same problems but other people had no problems at all.

Feedback from the FreeBSD community suggests that the problem is not unheard of, but also not universally reproducible.
So, I assume it must be some hypervisor misconfiguration?

I’m NOT the administrator of the KVM hosts. I can ask them tomorrow, though.

I’d like to get some ideas on what to look for on the hosts directly, if that makes sense.




Thanks in advance,
Rainer

Re: Question about KVM IO performance with FreeBSD as a guest OS

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1811 bytes --]

On Sun, Jun 23, 2019 at 03:46:29PM +0200, Rainer Duffner wrote:
> I have huge problems running FreeBSD 12 (amd64) as a KVM guest.
> 
> KVM is running on Ubuntu 18 LTS, in an OpenStack setup with dedicated Ceph-Storage (NVMe SSDs).
> 
> The VM „flavor" as such is that IOPs are limited to 2000/s - and I do get those 2k IOPs when I run e.g. CentOS 7.
> 
> But on FreeBSD, I get way less.
> 
> E.g. running dc3dd to write zeros to a disk, I get 120 MB/s on CentOS 7.
> With FreeBSD, I get 9 MB/s.
> 
> 
> The VMs were created on an OpenSuSE 42.3 host with the commands described here:
> 
> https://docs.openstack.org/image-guide/freebsd-image.html
> 
> 
> This mimics the results we got on XenServer, where also some people reported the same problems but other people had no problems at all.
> 
> Feedback from the FreeBSD community suggests that the problem is not unheard of, but also not universally reproducible.
> So, I assume it must be some hypervisor misconfiguration?
> 
> I’m NOT the administrator of the KVM hosts. I can ask them tomorrow, though.
> 
> I’d like to get some ideas on what to look for on the hosts directly, if that makes sense.

Hi Rainer,
Maybe it's the benchmark.  Can you share the exact command-line you are
running on CentOS 7 and FreeBSD?

The blocksize and amount of parallelism (queue depth or number of
processes/threads) should be identical on CentOS and FreeBSD.  The
benchmark should open the file with O_DIRECT.  It should not fsync()
(flush) after every write request.

If you are using large blocksizes (>256 KB) then perhaps the guest I/O
stack is splitting them up differently on FreeBSD and Linux.

Here is a sequential write benchmark using dd:

  dd if=/dev/zero of=/dev/vdX oflag=direct bs=4k count=1048576

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Question about KVM IO performance with FreeBSD as a guest OS

[rainer]

Am 2019-06-28 11:53, schrieb Stefan Hajnoczi:
> On Sun, Jun 23, 2019 at 03:46:29PM +0200, Rainer Duffner wrote:
>> I have huge problems running FreeBSD 12 (amd64) as a KVM guest.
>> 
>> KVM is running on Ubuntu 18 LTS, in an OpenStack setup with dedicated 
>> Ceph-Storage (NVMe SSDs).
>> 
>> The VM „flavor" as such is that IOPs are limited to 2000/s - and I do 
>> get those 2k IOPs when I run e.g. CentOS 7.
>> 
>> But on FreeBSD, I get way less.
>> 
>> E.g. running dc3dd to write zeros to a disk, I get 120 MB/s on CentOS 
>> 7.
>> With FreeBSD, I get 9 MB/s.
>> 
>> 
>> The VMs were created on an OpenSuSE 42.3 host with the commands 
>> described here:
>> 
>> https://docs.openstack.org/image-guide/freebsd-image.html
>> 
>> 
>> This mimics the results we got on XenServer, where also some people 
>> reported the same problems but other people had no problems at all.
>> 
>> Feedback from the FreeBSD community suggests that the problem is not 
>> unheard of, but also not universally reproducible.
>> So, I assume it must be some hypervisor misconfiguration?
>> 
>> I’m NOT the administrator of the KVM hosts. I can ask them tomorrow, 
>> though.
>> 
>> I’d like to get some ideas on what to look for on the hosts directly, 
>> if that makes sense.
> 
> Hi Rainer,
> Maybe it's the benchmark.  Can you share the exact command-line you are
> running on CentOS 7 and FreeBSD?
> 
> The blocksize and amount of parallelism (queue depth or number of
> processes/threads) should be identical on CentOS and FreeBSD.  The
> benchmark should open the file with O_DIRECT.  It should not fsync()
> (flush) after every write request.
> 
> If you are using large blocksizes (>256 KB) then perhaps the guest I/O
> stack is splitting them up differently on FreeBSD and Linux.
> 
> Here is a sequential write benchmark using dd:
> 
>   dd if=/dev/zero of=/dev/vdX oflag=direct bs=4k count=1048576


Hi,

you can read more about it here:

https://forums.freebsd.org/threads/is-kvm-virtio-really-that-slow-on-freebsd.71186/


I used

[root@centos ~]# fio -filename=/mnt/test.fio_test_file -direct=1 
-iodepth 4 -thread -rw=randrw -ioengine=psync -bs=4k -size 8G -numjobs=4 
-runtime=60 -group_reporting -name=pleasehelpme


Also on FreeBSD.


FreeBSD's dd doesn't have the oflag=direct option.


We tried the work-around described here:
https://www.cyberciti.biz/faq/slow-performance-issues-of-openbsd-or-freebsd-kvm-guest-on-linux/

But it doesn't really change anything. Most likely, the fixes hinted in 
the mailing-list have already gone into the kernel at this point.

The compute-nodes are running
  4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 
x86_64 x86_64 GNU/Linux


We saw similar behavior on XenServer 6.5 and real-world performance 
matched the (abysmal) benchmark-results we got.

The "normal" FreeBSD dd does about 1.1MB/s right now.

On my desktop PC (OpenSuSE Leap 42.3) (some i5 6500) with a cheap-ass 
256GB Samsung desktop OEM SSD, I get 55MB/s using the above dd command.

The servers we use are pretty high end SuperMicro servers, with Mellanox 
40GBit cards etc.pp.




Best Regards
Rainer

Re: Question about KVM IO performance with FreeBSD as a guest OS

[rainer]

Am 2019-06-28 11:53, schrieb Stefan Hajnoczi:
> On Sun, Jun 23, 2019 at 03:46:29PM +0200, Rainer Duffner wrote:
>> I have huge problems running FreeBSD 12 (amd64) as a KVM guest.
>> 
>> KVM is running on Ubuntu 18 LTS, in an OpenStack setup with dedicated 
>> Ceph-Storage (NVMe SSDs).
>> 
>> The VM „flavor" as such is that IOPs are limited to 2000/s - and I do 
>> get those 2k IOPs when I run e.g. CentOS 7.
>> 
>> But on FreeBSD, I get way less.
>> 
>> E.g. running dc3dd to write zeros to a disk, I get 120 MB/s on CentOS 
>> 7.
>> With FreeBSD, I get 9 MB/s.
>> 
>> 
>> The VMs were created on an OpenSuSE 42.3 host with the commands 
>> described here:
>> 
>> https://docs.openstack.org/image-guide/freebsd-image.html
>> 
>> 
>> This mimics the results we got on XenServer, where also some people 
>> reported the same problems but other people had no problems at all.
>> 
>> Feedback from the FreeBSD community suggests that the problem is not 
>> unheard of, but also not universally reproducible.
>> So, I assume it must be some hypervisor misconfiguration?
>> 
>> I’m NOT the administrator of the KVM hosts. I can ask them tomorrow, 
>> though.
>> 
>> I’d like to get some ideas on what to look for on the hosts directly, 
>> if that makes sense.
> 
> Hi Rainer,
> Maybe it's the benchmark.  Can you share the exact command-line you are
> running on CentOS 7 and FreeBSD?
> 
> The blocksize and amount of parallelism (queue depth or number of
> processes/threads) should be identical on CentOS and FreeBSD.  The
> benchmark should open the file with O_DIRECT.  It should not fsync()
> (flush) after every write request.
> 
> If you are using large blocksizes (>256 KB) then perhaps the guest I/O
> stack is splitting them up differently on FreeBSD and Linux.
> 
> Here is a sequential write benchmark using dd:
> 
>   dd if=/dev/zero of=/dev/vdX oflag=direct bs=4k count=1048576
> 
> Stefan




Hi,


on advice from my coworker, I created the image like this:

openstack image create --file ../freebsd-image/freebsd12_v1.41.qcow2 
--disk-format qcow2 --min-disk 6 --min-ram 512 --private --protected 
--property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi 
--property hw_qemu_guest_agent=yes --property os_distro=freebsd 
--property os_version="12.0" "FreeBSD 12.0 amd 64 take3"


This time, I got a bit better results:


root@rdu5:~ # fio -filename=/srv/test2.fio_test_file -direct=1 -iodepth 
4 -thread -rw=randrw -ioengine=psync -bs=4k -size 8G -numjobs=4 
-runtime=60 -group_reporting -name=pleasehelpme
pleasehelpme: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 
4096B-4096B, ioengine=psync, iodepth=4
...
fio-3.13
Starting 4 threads
pleasehelpme: Laying out IO file (1 file / 8192MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=1461KiB/s,w=1409KiB/s][r=365,w=352 
IOPS][eta 00m:00s]
pleasehelpme: (groupid=0, jobs=4): err= 0: pid=100120: Fri Jun 28 
15:44:42 2019
   read: IOPS=368, BW=1473KiB/s (1508kB/s)(86.3MiB/60005msec)
     clat (usec): min=8, max=139540, avg=6534.89, stdev=5761.10
      lat (usec): min=13, max=139548, avg=6542.68, stdev=5761.00
     clat percentiles (usec):
      |  1.00th=[   13],  5.00th=[   17], 10.00th=[   25], 20.00th=[ 
1827],
      | 30.00th=[ 3032], 40.00th=[ 4555], 50.00th=[ 5538], 60.00th=[ 
6718],
      | 70.00th=[ 8160], 80.00th=[10290], 90.00th=[13829], 
95.00th=[17433],
      | 99.00th=[25822], 99.50th=[28967], 99.90th=[37487], 
99.95th=[40633],
      | 99.99th=[51643]
    bw (  KiB/s): min=  972, max= 2135, per=97.21%, avg=1430.93, 
stdev=55.37, samples=476
    iops        : min=  242, max=  532, avg=356.10, stdev=13.86, 
samples=476
   write: IOPS=373, BW=1496KiB/s (1532kB/s)(87.6MiB/60005msec)
     clat (usec): min=13, max=46140, avg=4174.36, stdev=2834.86
      lat (usec): min=19, max=46146, avg=4182.13, stdev=2835.08
     clat percentiles (usec):
      |  1.00th=[   40],  5.00th=[   90], 10.00th=[ 1012], 20.00th=[ 
2008],
      | 30.00th=[ 2474], 40.00th=[ 3097], 50.00th=[ 3949], 60.00th=[ 
4555],
      | 70.00th=[ 5145], 80.00th=[ 6063], 90.00th=[ 7439], 95.00th=[ 
9110],
      | 99.00th=[13435], 99.50th=[15401], 99.90th=[20055], 
99.95th=[22152],
      | 99.99th=[36439]
    bw (  KiB/s): min=  825, max= 2295, per=97.26%, avg=1453.99, 
stdev=66.67, samples=476
    iops        : min=  206, max=  572, avg=361.90, stdev=16.66, 
samples=476
   lat (usec)   : 10=0.03%, 20=4.14%, 50=3.47%, 100=2.29%, 250=2.04%
   lat (usec)   : 500=0.06%, 750=0.51%, 1000=0.71%
   lat (msec)   : 2=7.38%, 4=22.88%, 10=44.07%, 20=10.86%, 50=1.55%
   lat (msec)   : 100=0.01%, 250=0.01%
   cpu          : usr=0.11%, sys=2.08%, ctx=83384, majf=0, minf=0
   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
 >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      issued rwts: total=22092,22436,0,0 short=0,0,0,0 dropped=0,0,0,0
      latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
    READ: bw=1473KiB/s (1508kB/s), 1473KiB/s-1473KiB/s 
(1508kB/s-1508kB/s), io=86.3MiB (90.5MB), run=60005-60005msec
   WRITE: bw=1496KiB/s (1532kB/s), 1496KiB/s-1496KiB/s 
(1532kB/s-1532kB/s), io=87.6MiB (91.9MB), run=60005-60005msec



Which is more or less half (or a third) of what I got on CentOS.

Still, that's not acceptable for us.

In any case, the documentation on the OpenStack website is outdated at 
best.



Regards
Rainer

Re: Question about KVM IO performance with FreeBSD as a guest OS

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4306 bytes --]

On Fri, Jun 28, 2019 at 03:51:04PM +0200, rainer@ultra-secure.de wrote:
> Am 2019-06-28 11:53, schrieb Stefan Hajnoczi:
> > On Sun, Jun 23, 2019 at 03:46:29PM +0200, Rainer Duffner wrote:
> on advice from my coworker, I created the image like this:
> 
> openstack image create --file ../freebsd-image/freebsd12_v1.41.qcow2
> --disk-format qcow2 --min-disk 6 --min-ram 512 --private --protected
> --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi --property
> hw_qemu_guest_agent=yes --property os_distro=freebsd --property
> os_version="12.0" "FreeBSD 12.0 amd 64 take3"
> 
> 
> This time, I got a bit better results:
> 
> 
> root@rdu5:~ # fio -filename=/srv/test2.fio_test_file -direct=1 -iodepth 4

I think iodepth has no effect here.  It applies to asynchronous I/O
engines like ioengine=libaio.  It's ignored for psync.

> -thread -rw=randrw -ioengine=psync -bs=4k -size 8G -numjobs=4 -runtime=60
> -group_reporting -name=pleasehelpme
> pleasehelpme: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=psync, iodepth=4
> ...
> fio-3.13
> Starting 4 threads
> pleasehelpme: Laying out IO file (1 file / 8192MiB)
> Jobs: 4 (f=4): [m(4)][100.0%][r=1461KiB/s,w=1409KiB/s][r=365,w=352 IOPS][eta
> 00m:00s]
> pleasehelpme: (groupid=0, jobs=4): err= 0: pid=100120: Fri Jun 28 15:44:42
> 2019
>   read: IOPS=368, BW=1473KiB/s (1508kB/s)(86.3MiB/60005msec)
>     clat (usec): min=8, max=139540, avg=6534.89, stdev=5761.10
>      lat (usec): min=13, max=139548, avg=6542.68, stdev=5761.00
>     clat percentiles (usec):
>      |  1.00th=[   13],  5.00th=[   17], 10.00th=[   25], 20.00th=[ 1827],
>      | 30.00th=[ 3032], 40.00th=[ 4555], 50.00th=[ 5538], 60.00th=[ 6718],
>      | 70.00th=[ 8160], 80.00th=[10290], 90.00th=[13829], 95.00th=[17433],
>      | 99.00th=[25822], 99.50th=[28967], 99.90th=[37487], 99.95th=[40633],
>      | 99.99th=[51643]
>    bw (  KiB/s): min=  972, max= 2135, per=97.21%, avg=1430.93, stdev=55.37,
> samples=476
>    iops        : min=  242, max=  532, avg=356.10, stdev=13.86, samples=476
>   write: IOPS=373, BW=1496KiB/s (1532kB/s)(87.6MiB/60005msec)
>     clat (usec): min=13, max=46140, avg=4174.36, stdev=2834.86
>      lat (usec): min=19, max=46146, avg=4182.13, stdev=2835.08
>     clat percentiles (usec):
>      |  1.00th=[   40],  5.00th=[   90], 10.00th=[ 1012], 20.00th=[ 2008],
>      | 30.00th=[ 2474], 40.00th=[ 3097], 50.00th=[ 3949], 60.00th=[ 4555],
>      | 70.00th=[ 5145], 80.00th=[ 6063], 90.00th=[ 7439], 95.00th=[ 9110],
>      | 99.00th=[13435], 99.50th=[15401], 99.90th=[20055], 99.95th=[22152],
>      | 99.99th=[36439]
>    bw (  KiB/s): min=  825, max= 2295, per=97.26%, avg=1453.99, stdev=66.67,
> samples=476
>    iops        : min=  206, max=  572, avg=361.90, stdev=16.66, samples=476
>   lat (usec)   : 10=0.03%, 20=4.14%, 50=3.47%, 100=2.29%, 250=2.04%
>   lat (usec)   : 500=0.06%, 750=0.51%, 1000=0.71%
>   lat (msec)   : 2=7.38%, 4=22.88%, 10=44.07%, 20=10.86%, 50=1.55%
>   lat (msec)   : 100=0.01%, 250=0.01%
>   cpu          : usr=0.11%, sys=2.08%, ctx=83384, majf=0, minf=0
>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>      issued rwts: total=22092,22436,0,0 short=0,0,0,0 dropped=0,0,0,0
>      latency   : target=0, window=0, percentile=100.00%, depth=4
> 
> Run status group 0 (all jobs):
>    READ: bw=1473KiB/s (1508kB/s), 1473KiB/s-1473KiB/s (1508kB/s-1508kB/s),
> io=86.3MiB (90.5MB), run=60005-60005msec
>   WRITE: bw=1496KiB/s (1532kB/s), 1496KiB/s-1496KiB/s (1532kB/s-1532kB/s),
> io=87.6MiB (91.9MB), run=60005-60005msec
> 
> 
> 
> Which is more or less half (or a third) of what I got on CentOS.

Are you using the exact same fio command-line on CentOS?

Have you tried virtio-blk instead of virtio-scsi?

Are you able to post the QEMU command-line from the host (ps aux | grep
qemu)?  Since --property os_distro=freebsd was used to create the guest
it's likely that the guest configuration is different from the CentOS
guest.  Let's compare the two QEMU command-lines.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]