Monitoring disk i/o in Windows

Monitoring disk i/o in Windows

[Elliott Balsley]

I built a Windows 2019 server with an NVMe storage pool and I'm using
fio to test performance.  Results from fio are in line with my
expectations for this hardware, but I noticed something strange with
Windows Performance Monitor.  The physical disk r/w throughput
counters are roughly double what they should be.  This holds true for
various different iodepth, numjobs, and block size options.  Logical
Disk throughput counters are accurate.  In this screenshot, fio shows
11.1GiB/s and the physical disk shows about 22.2 GiB/s.  Is there a
logical explanation for this?  Or is there a better disk monitoring
tool to use, something like iostat for Windows?

https://i.postimg.cc/zqrW8ZPk/windows-fio.png?dl=1

The storage pool is created as a simple stripe with no parity or
mirroring and formatted as ReFS.

Thanks,
Elliott

RE: Monitoring disk i/o in Windows

[Elliott, Robert (Servers)]

> -----Original Message-----
> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf
> Of Elliott Balsley
> Sent: Thursday, April 30, 2020 3:52 PM
> To: fio <fio@vger.kernel.org>
> Subject: Monitoring disk i/o in Windows
> 
> I built a Windows 2019 server with an NVMe storage pool and I'm using
> fio to test performance.  Results from fio are in line with my
> expectations for this hardware, but I noticed something strange with
> Windows Performance Monitor.  The physical disk r/w throughput
> counters are roughly double what they should be.  This holds true for
> various different iodepth, numjobs, and block size options.  Logical
> Disk throughput counters are accurate.  In this screenshot, fio shows
> 11.1GiB/s and the physical disk shows about 22.2 GiB/s.  Is there a
> logical explanation for this?  Or is there a better disk monitoring
> tool to use, something like iostat for Windows?
> 
> https://i.postimg.cc/zqrW8ZPk/windows-fio.png?dl=1
> 
> The storage pool is created as a simple stripe with no parity or
> mirroring and formatted as ReFS.

Writing to files means you're subject to however the filesystem
works. Maybe it's writing everything twice as part of journaling?
Accessing the raw physical device would be simpler.

It looks like fio's 1 MiB IOs are getting split into smaller IOs. Although
your LogicalDisk Avg Disk Bytes/Write is 1 MiB, the PhysicalDisk reports
419,422.101 bytes (which is just below 400 KiB).

If not just fundamental to the filesystem, it might be exceeding a limit
somewhere in the stack (e.g., too many scatter-gather list entries or
PRP entries). ~10 years ago we ran into a limit somewhere in the stack
of 255 max SG entries, with each entry describing one 4 KiB CPU page,
which was 4 KiB short of a 1 MiB transfer size. That kept 1 MiB IOs
from going through in Windows.